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ABSTRACT 

The report examines effective state monitoring 
policies in the education of handicapped children. Following a 
review, in part I, of the mandate in P.t. 94-142 z The Education for 
All Handicapped Children Act , for state education agency (SEA) 
monitoring, part 11 identifies SEA policies to monitor the quality of 
educational programs (Nebraska^ Missouri , North Carol ina A California, 
and Oregon). In part III, SEA monitoring of parallel state s agencies 
are described (Minnesota, Arizona, and Virginia^. The growth of 
interest in methodologies for .quality monitor ing is noted^ this 
interest exists despite diminished resources for conducting 
monitoring Each of the eight state approaches is detailed in the 
appended material. (CL) 
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The Education for All Handicapped Children Act, Public Lav; 
94-142, was enacted in 1975. The statute requires that a "free 
appropriate public education" be available to all handicapped 
children {age 3 through 21) in the United States, regardless of 
the severity of their handicap unless services to children aged 
3-5 or 18-21 would be inconsistent with state legislation. The 
law also mandates that. State. Education Agencies (SEAs j and 
Local Education Agencies (LEAs ) develop special education and 
related services to meet these children's Unique heeds. In 
tandem with Section 504 of the Rehabilitation Act, as amended, 
this law has had , and continues to have, a profound impact on, 
not only handicapped children and their families, but also the 
entire public education system. 

Implementation of P.L. 94-142 has proven difficult in many 
respects. While the law mandated major new respons ibi 1 i t ies to 
state and local education agencies, it did not provide detailed 
federal guidance nor full financing to carry cut these 
responsibilities. As a result, state and local education 
agencies have had to develop a wide range of new policies to 
implement the law. In so doing, they have confronted problem? 
and controversies ranging from the consequences of shrink ing 
human service resources and the debate over the rights of 
handicapped persons, to professional disagreements about the 
most effective settings In which to educate handicapped 
ch i ldren . 

Recognizing the importance of providing, states with 
technical assistance to implement ^ . L . 94-14 2 > Special 
Education Programs (SEP j of the U.S. Department of Education 
(formerly the Office of Special Education ) awarded a contract, 
to the Center for the Study of Social Policy (CSSP) to ( 1 ) 
identify effective policies used by state and local .education 
agencies that serve handicapped children; and [2 j disseminate 
information about these strategies to federal, state, and local 
dec is ion-makers . 

In conduct ing this project, the Center analyzed state and 
local policies in five areas of implementation: 

• Interagency co 1 la bora t ion ; 

• Provision of related services; 

• Provision of services to handicapped students in 
out -of-di strict placements ; 

• Implementation' of tho least restrictive environment 
mandate ; and 

• State monitoring and compliance activities; 
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The project design proceeded from a broad overview of policies 
and imp lenient a c ion strategies developed by s tates and local 
districts, through successive stages of data collection*. A 
telephone survey was conducted in all 50 states ; follow-up site 
visits were made to 18 states; and over 4 00 tEAs recommended as 
having off ective policies were surveyed, with approximately SG 
follow-up telephone interviews and field visits to some 35 
LEAs. 

From these data collection efforts, the project has 
produced four reports : 

Volume 1; Effective State Policies to Promote Interagency 
Collaboration . _ _ The first volume* sets forth a perspective on 
interagency collaboration which applies hot only to this volume, 
of the report, but to the other three volumes as well. This 
volume also reviews the use of state interagency committees , 
interagency agreements , and other collaborative efforts 
designed to ( 1 ) define responsibilities for services to 
children in residential facilities; (2) promote local inter- 
agency collaboration; (3) assign service delivery and financial 
responsibilities among state agencies; and (4) share 
information across agencies. 

Volume 2: Effective Policies in the Provision of Related 
Services . This report documents effective state and local 
policies in providing related services to handicapped children. 
The areas reviewed here include those state policies which 
clarify education agencies 1 responsibiii t ies , and those which 
'increase the resources available for related services by 
securing other state agencies 1 cooperation . This volume also 
examines local pol ic ies wh i cti (1 ) obtain resources from other 
human service agencies, (2) pool resources to increase the 
availability of services, and (3) seek to develop new programs 
for spec i f ic popu la t ion groups such as emotionally disturbed 
students . 





VbjLume 3: Policies Which address -aut-^oi^District 
P la ce ments and Assur e Educa t i on in the Le a s t Res^r^i^^ti-y^ 
Environment . Th is volume examines two important policy a re as : 
the provision of services to children in out-of-d is t r ict 
placements and the implementation of the least restrictive 
environment mandate. State policies are analyzed which help 
SEAs influence local placement decisions, as well as others 
which transfer responsibility back to the LE As for 
institutionalized handicapped students. This volume also 
examines local policies which utilize the resources of other 
human service agencies ;to implement the LRE mandate. These 
policies include those through which LEAs develop new programs 
to enable students to remain in local public schools; others 
that reflect LEA procedures to allow greater control over- 
placement decisions, arid still others that seek to change 
a 1 1 i tudes about integrating handicapped and non-handicapped 
students. 




Volume 4i Effective StaJte Monitoring Policies . The final 

volume examines two policy areas. Thefirst focuses on SEA 
policies that seek to evaluate program qua , 1 i ty as we 11 as 
perform compliance monitoring. The second examines alterna- 
tive strategies used by SEAs to effectively monitor education 
programs administered by other state human service agencies; . 

Support for this work was provided by Special Education 
Programs , . the U.S.. Department of. Education , under Contract 
#300-80-0829. Full responsibility for the accuracy of its 
findings and conclusions rests with the Center for the Study of 
Social Policy. However, many thanks are due to the officials 
of state and local education agencies and other human service 
agencies who gave their time to discuss their programs and pro- 
vide the information upon which the projects 1 reports are 
based. In addition, staff of the Center -would like to extend 
particular thanks to several people whose efforts contributed 
to these reports. Ray Smiches , the study's initial contract 
officer at the U.S. Department of Education, helped % def i he the 
scope of che study and contributed to its work throughout. 
David Rostetter and Jaddis Franklrn, the subsequent contract 
off icers , made numerous improvements i n the style and content 
of the reports . Dr. Kenneth Ol sen and Ethel Bright from the 
Mid-South Regional Resource Center, the University of Kentucky , 
generously shared their own work; assisted in the Center's data 
collect i efforts, and worked collaboratively in the prepara- 
tion of the related services volume. Dick Galloway and Beverly 
Os teen of the National' Association of State Di rectors of 
Special Education also assisted Center staff in all phases bf a 
the project f s work . 
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" VOLUME 4 

EFFECTIVE _STATE MONITORING POLICIES 
(Quality Monitoring arid Monitoring of State Operated 

EXECUTIVE SUMMARY 

One of the most challenging aspects of the Education for 
All Handicapped Children Act (P. L. 94-142 ) is the requirement 
that state education agencies ( SEAs ) monitor all education 
programs for hand i capped child re n within the ir state to ensure 
compliance with federal law. This provision broadened SEAs' 
responsibilities substantially, and while SEAs have made great 
progress in carrying out this mandate, they have encountered 
major difficulties as well. This report summarizes some of 
the lessons learned from SEAs 1 experience with monitoring , and 
then focusses on two policy areas which SEA officials identify 
as needing further work: 

• The need to monitor or evaluate the quality of 
special education programs; and 

• The need to monitor more effectively the education 
programs administered by other state agencies. 

I. THE MANDATE FOR STATE MONITORING UNDER P. L. 94-142 

The intent of the monitoring provisions in P.L. 94-142 is 
to assure a single line of accountability within a state for 
the education of all handicapped children. To this end, the 
federal statute required that SEAs monitor local education 
agencies ( LEAs } moire extensively; consequently, SF^As have 
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initiated three year monitoring . cycles to determine the extent 
to which LEAs comply with the policies and procedures 
established by federal and state law. In addition, P.L. 
94-142 required that SEAs monitor educational programs serving 
handicapped children that were administered by other state 
agencies . * This provision meant th^t SEAs^bad to review ^the 
educational programs of state-operated facilities arid 
i nst i tut iona 1 programs -- in many states , for the first time. 
The mandate thus put SEAs into a new, more difficult 
relationship with other state agencies. 

SEAs 1 experiences with the monitoring mandate already 



have generated both a number of lessons and several areas of 
concern. These include the following: 



• Compliance monitoring is one of the SEAs' most legiti- 
mate, and perhaps their most powerful, vehicle to 
influence LEA programming ; 

• However, SEAs have discovered that not all LEAs require 
the same level of mon i tor i ng . Once an LEA is in com- 
pliance, it may be more efficient to concentrate 
moni tor i ng elsewhere . 

• An increasing number of SEAs have concluded that most 
LEAs now need technical assistance as much as , or in 
addition to, compliance monitoring; 

• Officials in several SEAs argue that compliance 
monitoring efforts, at least as conducted in the past, > 
have overlooked some qualitative aspects of handicapped 
children* s educational programming ; 

• Progress in monitoring LEAs has not been accompanied by 
comparable success in monitoring programs in state- 
operated facilities. 

The knowledge that SEAs have gained from their monitoring 
experiences seems to be leading toward a concensus that (lj 
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compliance monitoring remains an essential tool to ensure the 
adequacy of local programs; (2) the time has come to consider 
building on; or revising, basic compliance monitoring systems 
to reflect more qualitative, dimensions of special education 
programming; and (3) the techniques which have proven success- 
ful in monitoring local programs must be adapted to ensure 
more effective monitoring of other state programs. 

II. SEA POLICIES TO MONITOR THE QUALITY OF EDUCATIONAL 
PROGRAMS 

The new and growing interest in measuring the quality of 
^special education programs reflects SEAs ' and LEAs ' desires to 
prove that their programs are effective. Budget constraints 
at all levels of government, in combination with mounting 
national criticism of public education, have caused SEA and 
LEA administrators to seek documentation of program quality. 

Progress in measuring quality has been slowed, however, 
by conceptual and operational difficulties. For instance, 
little concensus exists as to what constitutes quality, how to ^ 
assess it, and how to measure program outcomes. Fur!. her dif- 
ficulties result from the ambiguous relationship existing 
between quality and compliance monitoring . Should these two 
types of efforts be integrated Into one s y s > em, or can they, 
exist independent ly of one another? The:- - difficulties are 
compounded even further by the scarcity of appropriate 
materials designed to measure educational quality; thus* most 
* states undertaking this task must develop their own materials, 
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Despite these di.fficulti.es, a number of SEAs have 

^ • 

committed themselves to monito-ring the qe^_lit-y of special 

* . . ..... v V ^ , S . 

education p r og rams . Th"e e x p"e r i e rfce~~af foufs uch SEA s , aT. o n g 

with the experience of a state Mental Health Division, reveal 

some of the potential advantages as well as the problems in 

evaluating program quality: 



• The Nebraska Department of Education's ( N D E ' s ) 
. System for P ro g r^m Ef^f e ct i L ve ng s s Evaluation : 

Nebraska's system for evaluating program effec- 
tiveness grew f rom officials^ belief that it 
was time to move beyond compliance monitoring to 
assess the quality of special education ^programs 
statewide. A committee of educators convened by 
NDE developed a set of M service goals 11 which were 
based on committee members' perceptions of the 
characteristics of an effective special education 
program. These goa Is are the b a s i s for an 
evaluation ^Y 3 * 1811 ^ which will be field-tested by 
NDE in the fall of 1983 In 25 LEAs . 

• The _ M i s s o i ^....Dg^artren t _qf _E lernefit ar£-gfld 
S_ <e c o n d a jr£^]2 jdjj c 

E valuat ion : With the help of LEAs , the Missouri 
Department of Elementary and Secondary Education 
( DESE ) is developing a model to evaluate local 
special education programs and services in order 
to provide, state policymakers arid local 
administrators with information on the impacts of 
LEA programs. This evaluation system basically 
consists of a series of questions that probe the 
nature of an LEA's identification practices, IEP 
development, placement procedures, and other 
program characteristics. The system is self- 
administered by LEAs and will be field-tested 
during the 1983-84 school year. 

I nst ruction ' s Use of Program Ou^aJLj^t y Evaluation : 
The Division of Exceptional Children (DEC ) of this 
state 1 s Department of; Public instruction developed 
a system of . Program Quality Evaluation ( POE ) to 
assist_ldcal districts in assessing the effective- 
ness of their special education, programs. POE is 
a goal-based evaluation methodology which esta- 
blishes program goals arid objectives^ as well as 
detailed evaluation questions that will be Used to 
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determine* if objectives have been met, This 
system has. been designed to be of use to fcEAs and 
the SE A > i.e., contributing ihfortnation^ about 
local program quality . and state planning and 
program deve lopmeri t .. DEC 'eventually hopes to 
integrate PQE w.i t h' i t s. compi i arice monitoring 
system because of the bel ief t»hat one' consolidated 
monitoring system is both cost efficient and* 
e f f ecti\>e % c 

Th e _C a l^^or nia _D e £a £^m£ n^_o f^_^d u c a^o n ^£ 
I ntegrat ion— of Qu a lity Assessment a-nd Com p liance 
Moni torirtg : California's Department of Education 
is one of the few SEAs to have administered a 
monitoring system that wa,s designed .to combine 
compliance monitoring and measurement of program 
quality. This unitary system was built around a 
series of cr'i teiria statements against which school 
performance was measured. Whi le implementation of 
the system was reported to be- satisfactory , it is 
under revision as part of a general restructuring 
of the SEA's monitoring procedures. 

Progress Record System; Oregon T s Mental Health 
Division ( MHD) is evaluating 'the quality of 
programming for Trainable Mentally Retarded (TMR ) 
children in publ ic schools . (Under Oregon.* law , 
MHD is responsible for public school* programming 
for these children.) After obtaining extensive 
input from its consumers — tea cii§ r s — this 
division developed a system to measure student 
achievement , the Student Rfogress Record • (S PR ) , 
that focusses on individual children's progress, 
the SPR has been incorporated into IEP programs , 
is used to ensure program account abi 1 i ty , and has 
bolstered political and^i^ancial support. J ' 



As these examples illustrate, approaches to program' 
quality evaluation can differ markedly. However, the movement ,, 
toward measuring program qua 1 i ty seems to be gaining force. 
More and more SEAs are expressing interest in it, particularly 

/ 

as a way of documenting the value of their program for state 
and local public of f icia lsT^Most of these approaches demon- 
strate that SEAs 1 monitoring of state-operated program's is 
most effective when it is just one aspect of amor© 
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e n c ortipass i rig partnership between S-Efts ahd other state 
a^^neies- This report documents three of these approaches: 

• Th^Mtflfte g o h a _P^ga tfcjriej^_o f --|B3y.Ca t jo n ' s -Ose _o j 
I ntera^gnc^ Agrgggggts : The Minnesota Departmen t 
of Education eriteri^TTnto written agreements. with 
the Departments P E _ 5 Public Welfare Corrections 
to ensure that t_h®*t educational Programs complies 
with f^cieral and ?tate law. Both agencies also 
agreed to correct deficiencies cited by the 
mon i to t i ng tea^i Since the agreements wer^ 
signed^ the Department o f p u blic Welfa re 
facilities have b^e n brought into compliance by 
improving ^heir d Ue process, p^ tv acy protection, 
and s u trocjate Patent procedures. Greater 
difficulties were Encountered 1 h monitoring the 
correctional facilities, but substantial program, 
improvident has b e ^n made. 

• The_A ri Sgna^Pej^a r tm e nt^of JSd u cA^o^-£ -jjoni t orjjg 
of Yo^h Cor^ 1 jtj L es : As a result of 
several years _mori i tor i ng t he educational 
programs f or handicapped children i n this state's 
correCti cna i facilities, improvements have been 
made in t he Rind °E education incarcerated youth' 
have re Ceived. This improvement resulted from a 
strong -Working relationship between Department of 

; Education and Corrections 1 staff. The SEA pro-* 

vided intensive technical assistance; *ip-service 
training, and targeted additional funds to DOC 
-,* programs # a corrective action Pl a n is in place 
and continued progress is kging n *de. 

• The_ V iJP Jlja^P e p ^ ^^^^jlt^^ Education's u se of X^o re 
Monito^in^ Standards : - Working with four state 
agencie^ , the vi r gi. n i a Department of Educat ion 
developed a syste* of "core standards 1 ' for Inbej:- 
departmental btce^Ure and Certification. These 
core standards are_ administered by arts interagency 
team th ^t visits e ^0h e t a te-opera t e d residential 
facility. Due to the unusual degree of coopera- 
tion among different state agenci eS ; each state 
agency i s able^ to ^s^ist in reviewing children's 
total Programs , a nd , in that w^y, ensure the 
review's. substa n ti ve value arid efficiency; 

■' Virginia's approach stresses mee ting the basic 

requirements of federal and s t a t e education 
statutes in conjunction with the requirement s of 
other r e<s era \ and s tate human servi ce statutes . 
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EFFECTIVE STATE MOM I TORI N6 POLICIES 



INTRODUCTION 

One of the most challenging aspects of the Education for 
All Handicapped Children Act ( p . L * 9 4-142) has been the 
requirement that state education agencies ( SEAS j monitor all 
educational programs for handicapped chdldren within their 
state to ensu 4 ^ compliance with federal law. In most states , 
this mandate expanded the SEA monitoring role in at least two 
important ways. First, it required more extensive monitoring 
of local programs. Under P.L. 94-142 , SEAs have had to gather- 
much more detailed information about school districts 1 
policies and procedure's than .they did in the past. Second, 
the- statute broadened the scope of SEA monitoring: it 

required SEAs to review the educational programs of other 

» . _ _ _ ____ __ _____ __ ______ 

state agencies, thus placing SEA officials in the sometimes 

difficult position of evaluating the nature and quality of 

state operated programs which in the past had been completely 

autonomous . 

SEAs have made substantial progress in carrying out parts 
o£ their new monitoring responsibilities. For example, SEA 
officials indicate that monitoring of LEAs has proceeded well. 
Most state s have completed a three -year cycle of LEA moni- 
toring , and whivle: much work must be done before all LEAs are 
in compliance with federal law, administrators are optimistic 
that this goal/ can be achieved with the -aid of existing 
monitoring techniques. ' 

i 
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other aspects of SEA monitoring aire moire problemma t ic . 
Administrators cite two issues in particular which, they 
believe, will require careful policy development in the years 
ahead. These are: 

( 1 ) The need to monitor or evaluate the "quality" of 
special edueat ion programs ; and 

(2 ) The heed to monitor more effectively the education 
programs administered by other state agencies. 

Because state officials contacted in the course of this 
project voiced great concern about these two issues, they are 
the focus for this report on effective state monitoring 
pol ic ies . 

The report is organized into three sections. Section I 
reviews P. L. 94-142 1 s monitoring mandate in more detail and 
summarizes state experiences in implementing monitoring 
systems. Sections IT and III then analyze specific SEA 
policies that are related to the two issues identified above. 
Because many SEA policies in these two areas are experimental 
in nature, Sections tl and lit Contain a mix of developmental 
efforts and proven successes . tt may well be that the 
"emery ing" SEA policies will be most useful in illustrating 
the new directions that are open to SEAs . 



Ii THE MANDATE FOR STATE MONITORING UNDER P.L. 94-142 



The intent of the monitoring provisions in P.L; 94-142 
is to assure a single line of accountability for the educa- 
tion of all handicapped children within a state. To this end, 
the federal statute provides a broad mandate for state educa- 
tion agencies, requ i r i ng that they : 

"...provide for procedures for evaluation at least 
annually of the effectiveness of programs in meeting 

the educational needs of handicapped children 

(including evaluation of individualized educational 
programs), in accordance with such criteria that the 
Commissioner shall prescribe ... 11 
{Section 613. (a) (11) ) 

This mandate is made more specific in the federal regulations 

issued after the law's passage: 

"A general application must include assurances, 
satisfactory to the Commissioner. . . 

(3) That the state will adopt arid Use proper methods 
of administering each program, including: 

(ij Monitoring of agencies , inst i tut ions , arid/ 
organizations responsible for carrying out 
each program, and the enforcement of any 
obligat ions imposed on those agencies , 
institutions, and organizations under law; 

(ii) Providing technical assistance, if 

necessary , to those agencies , institutions, 
and organ izat ions ; 

( i i i ) Encourg aging the adoption of prom is trig or 
i nnovat ive educat iona 1 techniques by those 
agencies , institutions, and organizations; 

{ i v ) The dissemination throughout the state of 
i ri forma t iori on program requ i rements and 
successful practices; arid 

( v ) The correct ion of def iciencies in program 
operations that are identified through 
monitoring or evaluat ion. M 

(Education Division General Administrative Regulations, 



Section 100b. 101 (e) (3) ) 



Two aspects of this' mandate are particularly noteworthy. 
For one, the monitoring mandate under P;L. 94-142 is compre- 
hensive in scope. The SEA must monitor all programs in the 
state which receive public funds to educate handicapped 
children. This means that, in addition to monitoring LEfts and 
intermediate educational units (IEUs ) , the SEA must (1) moni- 
tor other state human service agencies that administer 
education programs for handicapped children, and (2) assure 
that any programs for handicapped children located in private 
facilities and paid for by public funds comply with the 

r 

requirements of federal law. 

Second, the regulations give prominence to the 
requirement that the SEA shall "undertake monitoring and 
evaluation activities to insure c omp liance. . .with the 
requirements (of the law)." To a large extent, this 
requirement has led SEAs to focus their monitoring efforts on 
the procedural aspects of federal and state law. The 
standards established to assess LEA programs have been based 
on the specific processes outlined in P.L. 9 4-142 -- e.g., 
procedures for I EP development, parent involvement, u$e of 
surrogate parents, etc. -- rather than broader issues 
concerning the nature or effectiveness of special education 
programs. Underlying this approach is an assumption that 
compliance with P.L. 94-142's specific requirements will 
c . assure quality programming. 

Since P.L. 9 4-142 1 s passage, SEAs have implemented 
monitoring systems designed to generate the information 
required by federal law and regulation. In fact, By now, 

o * is 

ERIC 



many aspects of these state systems have become similar. For 
example, studies have shown that SEAs' monitoring of local 
programs almost always incorporates the following stages: 

• Pr^- yisit pre p a rati dns , which include organizing 
staff , assigning responsibilities , and scheduling 
visits; 

• Q n -s i te procedur es , usiia 1 ly involving a team or an 
Tndi v i d u a 1 from the SEA reviewing LEA documentation 
procedures and examining i n d e tail a sampling of 
student records , particularly lEPs ; 

• Reporting , which usually, takes the form of immediate 
verbal feedback to LEA administrators at the completion 
of a monitoring visit and/ at a later point, a formal 
written report on the educational programs of the local 
educa t ion agency ; arid 

• Feedback and Follo w -up activities , which can include 
further SEA review of an LEA where deficiencies were 
noted, or correct ive action if the LEA agrees with the 
deficiencies and takes s t e p s " t o~" ire me d y ~ the m . 1 

Along with these similarities, state monitoring efforts 
differ in several ways as well. For example, the priority 
which SEAs assign to monitoring varies widely state by state. 
In some states, extensive staff resources have been devoted to 
this task, and the SEA has continually refined its monitoring 
materials arid procedures. Other SEAs have been content with a 
minimal monitoring effort -which they believe assures com- 
pliance with federal law but does not attempt to accomplish 
any broader purposes. 

SEAs 1 experiences in mon i tor i ng local programs already 

have generated a number of lessons. First, comp 1 i ance 

t 



1 John Tringo and Martin Kaufman^ "State Practices arid 
Progress in Monitoring P;L; 94-142, " unpublished. 



O - I? 

ERLC 



m onitoring is one of the most legitimate, arid perhaps the most 
powerful , vehicle that SEAs -have to influence _L E A pr ograms . 
There is rib doubt that vigorous SEA enforcement of federal arid 
state special education statutes has changed the nature arid 
scope of LEA programs. It has contributed to a widespread 
upgrading of local program standards and ensured that basic 
precepts of the federal law -- for example, the right to 

education in the least restrictive environment are 

\ 

i ncorpora ted into local programs. SEAs 1 compliance monitoring 
is one of the central reasons for the dramatic impact of P.L. 
9 4-14 2 nationwide. 

v 

Second, however, SEAs have discovered that not all LEAs 
require the same leveJLj5f monitoring . Initially, state pro- 
cedures were developed to provide uniform monitoring to all 
LEAs regardless of the quality or scope of their program. 
Naturally, smaller LEAs were monitored in less time and with 
fewer people than larger LEAs, but the basic procedures were 
the same for all. As SEAs have completed their first 
monitoring cycles, many have begun to differentiate among LEAs 
so that they can focus their mon i t or i ng efforts -- and 
particularly their f ol low-up or corrective action- efforts -- 
on LEAs where problems have been identified and spec i a 1 help 
is needed. SEAs argue that to do otherwise -- i.e., to 
continue to spread resources equally across all LEAs 
regardless of their situation -- is wasteful of increasingly 
scarce state personne 1 resources . 

Third, SEfAs are questioning whether - the t ime arid money 
spent in moni tor lug are yielding benef i ts of comparable ^va~l4ie . 
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Monitoring efforts consume a significant portion of state 
agency resources arid also impose burdens of time arid expense 
ori local education agencies. Voluminous data are collected^ 
and, depending on the state system, paperwork can be exten- 
sive. While no SEAs question the' need for some type of 

compliance monitoring, . an increasing number are asking if 

. _l - - . ■ 

current monitoring procedures fully meet the needs of either 

the SEA or LE As . Related to this point, some SEAs are 
considering whether the data collected through monitoring 
activities could be better targeted to meet specific informa- 
tion heeds of the SEAs. 

Fourth , an i ncreas i rig number of SEAs are cone lud i ng that- 
many LEAs now need technical assistance father than, or in 
addition to, compliance monitoring . Monitoring efforts of the 
past have inevitably carried with them an enforcement and 
oversight function. In fact, the nature of moni tor ing puts 
the SEA in a supervisory role, with an implicit threat of 
withdrawal or withholding of fyjnds if necessary. Rather than 
promoting LEA perceptions of the SEA as a partner in the 
effort to educate handicapped children, compliance monitoring 
efforts can genet ate an adversarial relationship between the 
SEA and local education agencies. to avoid this, many SEAs 
are attempting to give equal emphasis to technical assistance 
roles. In this context, the SEA views itself, and is viewed 
by the LEA, as a source of help in identifying LEA problems 
arid of advice and perhaps assistance in resolving these 
problems. This emphasis on technical assistance need riot 
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imply that compliance monitoring is unimportant (particularly 
* for LEAs not yet in compliance with the federal law), but it 
suggests that for some LEAS sufficient progress has. been made 
so that basic compliance issues should share priority with an 
ongoing attempt to improve and strengthen local education 
agency programs — a task that can best be done i f the LEA and 
SEA work together. 

Fifth , many SEA o f ficials are a r guing- that compl iance 
monitoring efforts, at least as they have been conducted in 
the past, have missed some fundamental, qualitative aspects of 
educa tJboria 1 programs fo r handicapped children , Compl iance 
monitoring's focus on the procedural aspects of state and 
federal law has tended to overshadow the qualitative aspects 
of educational programming. For example, it is comparatively 
simple for an SEA to observe whether an LEA has in place the 
' procedures necessary to develop an I EP and make a placement 
decision. The SEA can observe whether timelines are observed, 
whether parents are duly notified, and whether all written 
products are~in the student's file. However, an LEA can meet 
all these requirements and still be placing students in inap- 
propriate settings or failing to develop the resources in 
regular schools that are necessary to assure education in the 
least restrictive environment. A growing number of SEAs are 
now eager to move beyond looking simply at procedural arid 
quantitative measures of LEA performance and to begin investi- 
gating qualitative measures of educational programming as 
well. 
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Sixth* SEA officials realize that progress ir. monitoring 
idt-been -accompanied by comparable success iri ! moni- 
toring educational programs in state-operated facilities . I n 
part, this disparity is a matter of priorities; most SEAs 
have given greater attention to LEA monitoring because LEA 
programs serve the majority of children with handicapping 
conditions in a state. In part, however, the gap in progress 
reflects SEAs' inability to marshal 1 the resources necessary 
to effectively monitor the r&nqe of programs administered by 
other state human service agencies for handicapped children. 

In sum, the knowledge that SEAs have gained from their 
experience In monitoring local programs seems to be leading 
toward a consensus that (1) compliance, monitoring remains an 
essential and powerful tool to ensure the adequacy of local 
programs; (2) it is time to consider building^ on , or revising, 
basic compliance monitoring systems to reflect more qualita- 
tive dimensions of special education programming; and (3) the 
techniques which have proven successful in monitoring local 
programs must now be adapted to enable more effective 
monitoring of other state programming. 

Based on this consensus, two policy questions emerge as 
r Quiring attention by federal and state administrators as 
SEAs c>xitinue to develop their monitoring policies: 

l\,How and to what extent should. SEA monitoring 

efforts attempt to assess quality of education? 

2 . How Vn SEAs effectively monitor parallel state - 
agenc id^? 



The remainder of this report explores policies related to 
these two issues one of which represents the emerging 
"state-of-the-art" of state monitoring activities> and the 
other of which represents SEAs 1 on-going attempt to carry-out 
the "general supervision" mandate of P.L. 94-142. 
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II. SEA Rf 



:E g TO MONITOR THE QUALI TY OF R BtJCATlONAL 



PROG RAMS 

The issue of quality monitoring has grown in importance 
as state and local education agencies feel the need to gain 
more from their monitoring activities: more than descriptions 
of program procedures; more than just assertions that special 
education is somehow "working;" more, even, than information 
about LEAS' compliance with state and federal laws. Special 
educators, like all education off i-cials, are feeling the 
pressure of public scrutiny and, increasingly, are seeking to 
' ' prove that their programs are effective and of high quality. 

This has led to a series of efforts, undertaken independently 
• ' by a growing number of SEAs, to develop methods for measuring 

the quality of LEA programs. 

Progress in measuring quality has been slow,' however, 
because Such measurement is inherently difficult, with 
pitfalls that are both conceptual and operational in- nature. 

A number of factors make quality monitoring problemmatic. 

* ■■ ....... 

Of these, the first is the most fundamental: the concept 

of quality in special education is only beginning to be 
defined. The question "What constitutes quality?" can be 
answered in many different ways, and there*- is., as yet, no con- 
sensus within the special education field about the dinens ions 
that make-up quality programming. Similarly, the approaches 
: • to assessing quality are as Varied as the goals which oif- 

» _ 

•ferent people place' on special education programs. Should 
quality bo' judged by looking primarily at program design or 
structure? Or are program outcomes the only valid measure? 

1 1 
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Are cost factors relevant to assessing quality, or should 
evaluative judgements be made free of resource considerations? 
Is it necessary to closely examine student achievements, or 
can parental satisfaction be a sufficient proxy for assessing 
quality? Ultimately, the question is, whose" goalr; for special 
education are to be used when quality is assessed: those of 
policymakers, administrators, teachers, students, parents,. or 
some combination from ail these persons? 

r. 

SEA attempts to measure program quality are complicated 

___ ____ _____ _ _ _____ __ _______ _ ______ __ 

by'the need to define their relationship to compliance 

monitoring. Should these two types of efforts be integrated 

into one system, or can they exist independently of one ^ 

another? While the role of the SEA in compliance monitoring 

is clear, what should be the SEA role in quality assessment? y 

Perhaps most importantly, do SEAs have the resources to do a 

thorough job of both compliance monitoring' and quality 

assessment? Or does the movement toward measuring quality 

necessarily involve less intensive compliance monitoring? 

Those arid similar questions arc crucial as SEA officials think 

about measuring program quality. 

The policy problems involved in assessing program quality 

are compounded by the scarcity of materials that are designed 

for thi,s purpose. Most of the materials available to assess 

special education quality fall within one of two extremes ; 

They are either very general i.e., the^ set forth an t 

overall approach to quality measurement — in which case 

cons iderab le - work must still be done to tailor them to the 
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specific needs of ah SRA. Or they t§rid to b"qf highly specific, 
designed to measure a discrete program arid adopted to that 
program's goals. in the latter case, the materials are likely 
to be too one-dimensional to be immediately useful to ah SEA. 
Thus as most SEAs have found , the decision to med!sure program 
quality usually includes a commitment to develop appropriate^ 
materials as well. 

Finally, any attempt to measure quality may . encounter 
resistance from these whose work is being measured. By its 
nature, evaluation carries with it the risk that results will 
be unfavorable rather than favorable to program administra- 
tors, teachers, and others involved in education. While 
everyone endorses the concept of quality measurement, the 
reality of it can be intimidating for those whose "quality" is 
be ing assessed . = 

For all of these reasons, SEA attempts to measure quality 
involve more than a simple expansion of existing activities. 
They require- a process by which educators reexamine t h-e i r 
goals and then sefek to determine how well they are doing 
against these goals. Quality measurement also requires the 
participation of a wide range of people , in order to gain 
support from all those whose work will be assessed. Thus, 
when an SKA begins the process of quality measurement, the 
Ultimate benefits may hot materialize for several years and 
are almost certain to require the coordinated efforts of many 
people over an extended period of time. 
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Despite the formidable challenges, a number of SEAs have 
c^mitted themselves to monitoring or evaluating the quality 
o E special education program 3 • Ttjese efforts go By many dif- 
f^e n > names, and the n cRnen^h/ure alone i n this field has 
b^orne a source °f confusion* For purposes 05 clarification, 

generic t e r m " p f Q g r a s quality evaluation" is used 
f^^quently in the discussion ^hich follows r but it js meant to 
i n °lude the range 0 f qual i ty^eiated efforts which SEAs have 
undertaken. 

Four SEA efforts we examined are sufficiently developed 
that their experiences, and the ^roaches they have chosen, 
a** instructive at >o u t the problems an d potential for program 
g u ^lity evaluation, ' y m additio^, a n evaluati 0n s y St em used by 
a state developmental disabiiit i es agency to a s s ess the pro- 
g^ss'of trainable mentally re t a rded ( T M R ) s t u dents in local 
s^hoo^ progc^ms Provides another perspective on the measure- 
ment of quality.. The five efforts are exanfcned in turn below. 
Th e y are: t 

■-\ 

• Nebraska 1 s d e velc/pment of a system for program 
Effect ivene^ s Evaluation; 

• Missouri 1 * 5 System for Evaluating the eegeefc iveriess of 
special eHUo^tion proU^ms; 

• North Carolina's use of ~ Program Quality Evaluation to 
assess qualify D f ioc^l programs? 

• California's i nnorpor 3 t i Qn 0 f qual i ty measurement into 
its compliance monitoring system; and 

• Oregon's ^(5 of a Student Progress R^Qord to document . 
the effectiveness of * c h_nol programs fo r 'pMR students. 
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A; The Nebraska Pepartmen t-jo£ Education's System of 
Program Effectiveness Evaluation 

The Nebraska Department of Education is developing a 
system of prog ram evaluation whi : ch is designed to meet the 
SEA's needs for evaluative information^ as well as the compar- 
able heeds of LEAs • 



The Special Education Branch of the Nebraska Depart- 
ment of Education ( NDE ) has worked for several years 
to develop a metho'd of determining the effectiveness 
of special education programs. S-EA administrators , 
as well as LEA officials, have sought such a method 
because they felt it was time to move beyond com- 
pliance monitoring and assess the quality of special 
education prog rams statewide. In addition , NDE ' s 
effort responds to a 1973 state law that requires NDE 
to study the effectiveness of local special education 
programs. - 



Development of evaluation methods was begun in 1980 
when NDE entered, into a service agreement with the 
Midwest Regional Resource Center (MRRC) to obtain 
technical assistance on program effectiveness • In 
the following year, NDE convened a Program Effec- 
tiveness Development Committee , whose twenty members 
included LEA staff, higher education personnel, and 
special educators and general educators drawn from 
all geographic areas of the state. The Committee 
developed a set of standards -- or "service goals" 
which were based on the Committee members 1 percep- 
tions of the characteristics of an effective special 
education program. These standards covered program 
areas such as Administration; Service Delivery; 
Curriculum/ Ma terials/Facili t ies; Communication with 
Consume rs ; Fiscal Management; and Systems Aspects 
(i.e., Relationships between LEAs and other human 
service agencies). To obtain wide input on these 
standards, NDE then convened a symposium on Program 
Effectiveness Evaluation involving approximately 
educators from around the state who (1) reviewed and 
revised the standards ; (2) identified the types of 
information. that an LEA. would need in order to evalu- 
ate each of the service goals; and ( 3 ) identified 
potential pitfalls within each service goal; 



Rased on this revised set of service goals , approxi- 
mately 25 LEAs will field test the evaluation system 
in the fall of 1983. In some "districts , NDE's appli- 
cation of the evaluation system will be combined 
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with the regular monitoring visits of NDE staff; in 
other districts, NDE staff will just conduct . Program 
Effectiveness Evaluation. Prior to the field tests* 
NDE staff will receive further training to ensure 
that they have the skills to assist LEAs with proper 
evaluations . 

NDE officials are now working toward making the eval- 
uation data useful, for the state agency as well as 
LEAs* while still retaining flexibility in the „ 
evaluation methodology.. (For a more detailed des- 
cription of Nebraska's evaluation system, see 
Append ix A. ) 

Nebraska's efforts illustrate several of the factors 
that are moving SEAs toward program quality evaluation. The 
Department of Education began this work because officials 
there believed that the qualitative d imens ions . of special 
education were not adequately assessed or documented by 
existing monitoring procedures. They, like officials in other 
states, saw the need for a new type of evaluation effort that 
would help set standards for high quality programs, rather 
than simply assess LE As for compliance with federal law. 

Nebraska's efforts are also similar to those of other 
states in th,at NDE ' s work was partially motivated by legisla- 
tive demand for more information. While Ne braska may be 
atypYcal in haying this legislative interest expressed* in law 
(the 1973 statute), SEA officials' desire to respond more 
fully to legislative inquiries is not"unusual . 

Nebraska's methodology for Program Effectiveness 
Evaluation is also worth noting, because it illustrates a type 
bt" evaluative effort which is being followed or considered by 
other SEAs. _The "service goals" developed in the Nebraska 
system are an attempt to describe the characteristics of a 
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high quality — arid thus effective program. For the most 
part, these "service goals' 1 describe program attributes, i.e. f 
the types of policies, procedures, arid activities that can be 
expected to produce program quality. They do riot focus oil 
program or student outcomes, but assure that if the desired 
attributes of a program — the inputs, in evaluation terms 
are in place, then desired outcomes will be promoted. In this 
type of system, the concept of program quality is closely 
related to the concept of effectiveness, but is not identical 
to it. As will be noted in subsequent examples, other states 
have f ocussed more directly on program outcomes as a means of 
measuring program quality. 

Before discussing other aspects of Nebraska's Program 
Effectiveness Evaluation system, it is useful to look at a 
second SEA whose work both resembles and differs from the 
Nebraska ef f ort . 

B . The Missouri Department of Elementary— and Seco nxiar^ 
Education's System for Program Evaluation 

Missouri's approach to assessing the quality of 

special education programs is based on an evaluation system 

which would be self-administered by LEAs . 

The Missouri Department of Elementary arid Secondary 
Education (DESE ) is developing a model to evaluate 
local special education programs and services. A 
< primary goal of the model, is to provide local 
administrators with information on the impacts of 
their programs, but the information is intended to be 
Useful to state policymakers as well. The DESE 
approach to evaluation is oriented to LEA needs 
partly because much of the impetus for developing ari 
evaJLua t ion system came from LEAs themselves . 
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Missouri's evaluation, system, basically consists of a 
series of questions that probe the nature df_ an LEA 9 s 
identification practices, evcluatibh or diagnosis 
practices, IEP development -,_ placement procedures, and 
other dimensions of the local special education 
program. The evaluation adopts a systems approach 
that enables ah LEA to examine each stage of the 
major activities necessary for effective. program- 
ming. The model is self-admirii s tered by LEAs arid 
they in turn are responsible for identifying program 
areas that require improvement. 

The specific questions used in this system emerged 
from a series of meetings between SEA and LEA staff 
from February to June, 1983. State officials thus 
believe that the questions represent a professional 
consensus oil the factors necessary for effective 
programming. Missouri hopes to f ie4d test the system 
during the 1983-84 school year. DESE views this 
field test as a cooperative "pilot" effort, after 
which SEA and LEA staff will refine the model and 
j udge its Utility for LEAs . 

Both DESE arid LEA officials believe two benefits will 
result from the evaluation activity. Local districts 
will be able to Use information about program 
1 effective ness to gain continued support of their 

programs from local School boards. When local boards 
are faced with tough budget decisions, special 
education administrators realize that data on program 
; accomplishments are essential. In addition, DESE 

I staff stress that the evaluation is ^expected to _lead 

to program improvements. As stated in the DESE 
"Evaluation Review" document, "information on 
...programs and services.. i__ should result in action 
to improve them." jFor a more detailed description 
of Missouri's evaluation model, see Appendix B. ) 

i 

As! Missouri f s evaluation effort illustrates, the demand for 

I . . . ■ , 

ef |f dirts to assess educational quality may emerge from 
education practitioners at all levels of government. Program 
quality evaluation is not being imposed on LEAs by state 
administrators. Rather, LEA officials in Missouri, as in 

other states are often eager for it, recognizing that it 
meets their own information needs. 
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The Fact that quality assessment is often a mutual 
goal of state and local officials is well- i Illustrated by the 
collaborative mariner in which the Missouri system has been 
developed. there, as in Nebraska, L E As contributed to 
formulating the questions and will be an ongoing part of 
judging whether the system works. 

Missouri's activities also illustrate one of the 
major reason why quality measurement is becoming more desired. 
As was mentioned previously, special educators recognize the 
need to defend their programs, especially when education funds 
local tax revenues, as well as state and federal subsidies- 
face competition in a world of dwindling dollars. Special 
education's growth in the past decade resulted not just from 
the P;Li 94-142 mandate, but from a widespread conviction that 
its program provided clear and direct benefits to handicapped 
children. Now that new programs have been created and special 
education's rapid program' growth has slowed somewhat, admini- 
strators must be able to justify continued investments. Most 
administrators are confident they can make a strong case if 
data on program quality and effects can be more systematically 
produced. 

As noted, Missouri's efforts on program evaluation 

aire similar in . some ways to Nebraska 's* in par J:.,. this is due 

to common elements in t" ;, n Movolopment of the two states' 
activities: both SEAs involved the MRRC in their early plan- 
ning; they both reviewed some of the same materials in 
developing their systems; and administrators from Missouri 
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attended Nebraska's early meetings about quality assessment. 
But in addition, the similarities between the two states 
represent patterns that will probably occur in most SEAs as 
they begin to move toward program quality evaluation. 

For example, in both states, the- SEA decided to take 
a leadership role even though much of the desire for evalua- 
tion systems was expressed by local administrators. In both 
Missouri and Nebraska, SEA officials realized that the task of 
developing evaluation standards was probably beyond the 
capacities of any one LEA. Without SEA coordination, program 
quality evaluation was unlikely to materialize. This is 
liable to be true in most states. 

Similarly^ develop r. t e n t of materials oh quality 
evaluation has been ah inductive process in Nebraska and 
Missouri, and would probably be so in any state. Neither 
Nebraska nor Missouri simply adopted a preexisting set of 
materials as the basis for their evaluation; appropriate 
pre-packaged materials do not exist. Because the state-of- 
the-art in program effectiveness evaluation is still 
rudimentary, SEAs moving in this direction wilt have to devote 
much of their effort to developing materials. 

Even if evaluation materials were more generally 
a v ai lable , . Nebrask a ' s arid Miss ou r i 1 s expe r ien c e.s. „.ind i c a te t hat 
tailoring the methodology to a state's specific needs is 
essential. This process forces SEAs to be explicit about the 
purposes of evaluation, tho audience for evaluation results, 
and the precise information that will be generated by the 
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evaluation. In addition, the process of developing evaluation 
methodologies also serves to marshal 1 support. SEAs* and 
LEAs 1 collaborative work in developing either the evaluation 
questions (as in Missouri) or service goals (as in Nebraska) 
reduces the threat inherent in evaluation for all partici- 
pants, it ensures that an evaluation will riot be seen as 
something imposed on LEAs by a higher level of government, but 
rather as a useful tool which the LEAs themselves have 
designed. 

C . The North Carolina Department of Public Instruction's 
Use of -P-roqram Quality Evali 1 ^ t ion 

Several of these same elements can be observed in yet 
a third approach to quality evaluation. While similar to the 
activities just described, North Carolina's approach differs 
in several important ways as well. 

The Division for Exceptional Children (DEC) of the 
North Carolina Department of Public Instruction has 
developed a system of Program Quality Evaluation 
( POE ) to assist local districts in assessing the 
effectiveness of special education programs. 

As in other states , DEC officials began this effort 
because they felt that compliance monitoring focused 
primarily on the " p r%c ess" of special (education. 
What was needed, fchsy believed, was to supplement 
such monitoring with "program" monitoring that would 
determine the quality of programs for handicapped 
children. The system has been developed to answer 
basic questions such as: Is a special program 
effective? Are special needs- c h i 1 d r e n learning 
academics and skills? Which parts of the special 
education pro <j ram are excellent and which are 
satisfactory? 

DEC contracted with APT, Inc., from West Hartford, 
Connecticut, to develop evaluation materials and to 
train DEC staff: in evaluation methodology. After 
obtaining extensive input from DEC 1 s Regional 
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Coordinators, as well as from other DEC and LEA 
staff, APT f_i __e Id tested th e Prqgra mQuality 
Evaluation (POE) model in two school systems, 
Revisions were then made for the final system manual; 

POE is a goal-based evaluation methodology which 
establishes, first, a series of broad goals con- 
cerning the program to be evaluated? second?- specif ic 
program objectives related to these goals; and third, 
detailed evaluation questions which determine if the 
objectives have been met. The objectives and ques- 
tions focus on prog ram operations and numeric a 1 
criteria are established as benchmarks of satisfac- 
tory performance, (For example, a sample goal might 
be that "75% of students in a program attain desired 
reading sk ills . " ) 

DEC is now moving toward broader implementation, and 
will be awarding up to sixteen competitive incentive 
grants, of from $2,000 to $10 , 000 to encourage local 
units to participate irv POE in the 1983-84 school 
year. The local agencies using POE will be exempt 
from a program compliance visit or a self-monitoring 
* report — the other forms of monitoring and 

evaluation which DEC uses to assess LE As • (For a 
more de tai led _descr ipt ion of North Carolina's system, 
see Appendix C . ) 



Several aspects of North Carolina's POE system are 
particularly interesting. First, it was designed' to be of 
direct assistance to the SEA as well as to LEAs. As in the 
case of Nebraska' 1 s and Missouri 1 s systems, POE is intended to 
contribute to public information on special education (and 
thus' build community and po-l it ical support ) and to improve 
local program quality. On this latter goal , DEC makes clear 
that local school districts using POE are expected to remedy 
any def iciSKc ies which are- found" throuqh POE procedures - In 



addition to assisting LEAs, however, POE is designed to 



improve state planning and program development. DEC hopes to 
ag-jregate evaluation data from many LEAs to obtain an improved 
picture of the quality of special education statewide. 
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Specifically/ DEC believes that POE will eventually allow the 
state to: 

* develop a state-wide perspective on special education 
. and budgetary needs; 

# produce long-range plans; and, 

• make recommendations to the State Board of Education 
arid the General Assembly. 

A second interesting facet of North Carolina's system 
is that DEC intends, to integrate POE with its compliance 
monitoring system. DEC makes clear in its written materials 
on the system that POE does not substitute for federal and 
state compliance monitoring; the two systems are thought of as 
two parts of one comprehensive monitoring system, thus 
avoiding the duplication of effort which can occur when such 
systems are developed in a parallel) but uncoordinated, 
fashion . 

Finally, North Carolina's approach to broader 
implementation i . e , using a competitive incentive grant 
process recognizes that LE As may incur extra costs in 

implementing any hew system. Implicit in DEC ' s willingness to 
pay for some of these initial costs is a recognition that 
LEAS are more likely to implement this system if they receive 
help financing it. 

North Carolina's interest in integrating program 
quality evaluation with compliance monitoring has been shared 
by other SEAs that have considered more extensive measureoient 
of quality. -One rationale for such consolidation is that the 
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resulting monitoring system* is more efficient. Moreover, it 
can be argued that the distinctions which are sometimes drawn 
between program quality evaluation arid compliance monitoring 
are spurious. these are actually just differing approaches to 
th^' same goal, i.e., improved educational programs for handi- 
capped students. the next example illustrates advantages arid 
d is ad vantages of such integrat ion . 

D. The California Department of Education' s I ntegration 
of Quality Assessment arid Compliance Moftj^ta^irig 

The California Department of Education is one of the 

few SEAs to have actually implemented a monitoring system 

combining compliance and quality monitoring. Although 

California's system is now under revision, it provides an 

example of how quality assessment can be made an integral part 

of a monitoring system. 

In the school years 1 9 8 1 - 8 2 and 1 9 8^2-8 3,, the 
California Department of Education used monitoring 
procedures which included both compliance and quality 
measures. As in many other states, the Department 1 s 
emphasis. bri quality reflected wide-spread sent iment 
among LEAs that compliance monitoring, as it was then 
done, provided little assistance to LEAs trying to 
improve their programs. 

Department officials chose to modify their overall 
^ monitoring system to include quality measures rather 

than develop a parallel system that would assess 
program effectiveness.; Their goal was to emphasize 
examination of the total program for handicapped 
students, and they felt that both compliance arid 
quality measures contributed to this aim. 

the methodology for the California system was derived 
from two sources: (1) the School Improvement 
Monitoring System, a child- centered evaluation system 
for all: educational programs, which was already in 
use -in selected districts in the state; and (2) the 
compliance monitoring system that the Special 
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Education Division had been using. The new system 
was bui It around a series of criteria statements ( in 
effect, goal statements for effective practice). To 
measure school performance against these criteria a 
series, of more detailed questions were set fbrth^ 
some of which were designed to be asked of admini- 
strators^ some of parents, some of teachers* and some 
to be answered through document review. A proportion 
of these (questions addressed compliance issues and 
were tied to specific provisions of state or federal 
law. Other's assessed school performance or children's 
progress on dimensions not required By law, but 
judged as important • These quality questions had , 
for the most part, Been _developed by SEA staff with 
a s s is t a nee from f i e id pe r s onne 1 (tea c her s and 
administrators) who were consulted in the process. 



The system was tested in school year 1 981-82 and 
revised for 1982-83. Field evaluations of the second, 
year were extremely positive and SEA staff believed 
the system gave a comprehensive view of the 
performance of local special education programs. 

All of the monitoring systems of the California 
Department of Education aire now under ^revision , due 
to a hew effort to forge one integrated school moni- 
toring system across all educational programs . As 
part of this e f fort , monitoring systems are being 
reduced pr imar i ly to their compl i ance components . 
Thus, many of the qual i ty d imens ions incorporated in 
the special educa t ion monitoring sy s t em of the past 
two years may be discarded despite their ef f ect ive- 
: ness . ( For examples of California's procedures , see 

Appendix D. ) > ■ 

California's System to assess program quality has 
several advantages which, even though no longer used in 



California, may be useful to other SEAs . 

California's integration of compliance monitoring and 
quality evaluation seemed effective both concep t Ma 1 ly and 
practically. Conceptual ly , this integration ref lec ted the 
conviction that the two types of mon i t or i ng should be linked 
because, ultimately, both seek to determine the degree to 
which programs' are serving students well. This integration 
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also had offered practical advantages. School districts had 
to cope with only one system that united compliance moni- 
toring and quality evaluation criteria, not two, As a result, 
SEA visits to school districts were consolidated, and training 
for monitoring, personnel cbiild combine both quality arid 
compl iance procedures . 

California's specific methodology for consolidating 
compliance and quality "criteria" also offers some lessons to 
other S E A s . Too often, systems that evaluate educational 
programs can devolve into a long series of unrelated ques- 
tions, with the focus being lost. By weaving compliance and 

quality-assessment concerns- into a se.Lo.f unified, criteria, 

California's system seemed to maintain a clear focus on the 
evaluation's overall purpose. This clarity can be critical in 
determining whether a system is accepted by administrators, 
teachers, arid parents. j 

California's experience with this system points to 
two additional aspects which SE As contemplating program 
quality evaluation should study. California's system requ ired 
significant state investment because it was comprehens i ve 
-validated its findings by cross- interviews and record reviews 
f rom a wide range of sources , arid because field personnel 
conducted the mcSnitoring review. California officials 
indicate that cosrs would be cut if state staff were used for 
these purposes. Also, because all LEAs used ail parts of the 
system, it was time-consuming for SEA and LEA staff. Finally, 
in Ca 1 if ornia -this type of system was mandatory for all LEAs; 
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states seeking to reduce the SEA's prescriptive authority over 
LEAs might choose to make parts of such a system voluntary. 



E . The Oregon Mental Health Division's Student Progress 
Record System 

Oregon ha-s taken a quite different approach to 
evaluating the quality of special education programmi rig . 
Because of an unusual historical development of special educa- 
t ion programming in Oregon, the Mental Health Division (MHD) 
in the State's Department of Human Resources oversees local 
school districts* programming for TMR children. Therefore, 
the MHD agency, and not the Department of Education, is 
evaluating the quality of TMR programming. The Oregon system 
emphasizes detailed measurement of each, hand i capped child's 
educational progress. By aggregating this information, the 
state agency is able to judge broader program effectiveness. 



The Oregon Mental Health Division has implemented a 
system of student progress evaluation that measures 
the quality of education received- by Trainable 
Mentally Retarded (TMR) children in Oregon public 
schools. (In Oregon, the TMR designation includes 
moderately, severely, and profoundly retarded 
children. ♦ 



Use of this system began shortly after 1969, when the 
state legislature first assigned respons i b i 1 i ty for 
educating TMR children to the Division of Mental 
Health. Prior to that time, most of these children 
had been living at home or in an institution arid 
there was considerable skepticism among educators 
as well as parents that they could be adequately 
educated in public schools. thus, as MHD officials 

^Under current Oregon law, the Division of Mental Health 
is responsible for educational programming for TMR children/ 
through contracts with local school districts . Of 
approximately' 1, 400 TMR children in the state, all but 50 
are being educated in publ ic_ schools as part of the programs 
set up by the MR/DD Program Office, Mental Health Division. 
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supervised the movement of these children from other 
.settings to publ ic. schools j _they felt it was critical 
to (1) show that these children, could make progres 
in public schools* and (2) es t a bl i s h _accbiiri ta b i 1 i ty 
over local school programs for the effectiveness of 
the instructional: programs. 



MHD found its first system for measuring student 
achievement, based on three standardized tests, 
unsatisfactory . Subsequently, MHD called together 
special education "teachers to review fundamental 
questions , such as "what were chi ldren being taught?" 
and "what should children be expected to know after 
each teaching year?" The instrument which was 
eventually developed was tested for several years, 
beginning in 1971. It was substantially revised in 
1975 and again in 1978 in response to the strengths 
and weaknesses identified by its primary users, the 
teachers. For example, in the 1 978 r e v i s i on, 
teachers criticized the instrument's length and its 
inability to measure small increments of progress . 

A-s—a— re-su It of this critique, the system was modi- 
fied; additional questions were developed, and the 
instrument was divided into ski Unrelated sub-units. 
Teachers were instructed to use only the appropriate 
sub-units , thereby gaining more detailed information 
on a child, while reducing the testing time 

v involved. 

Over time, the Student Progress Record (SPR) has been 
used in many .ways . In its early years, its results 
were used to show the legislature and the public that 
'TMR children were making progress in the public 
schools and thus to justify continued program 
support. More recently, SPR has been ah integral 
part of the IEP process, establtshing very clear 
expectations for' the development of each child. 

MHD also used the SPR to ensure program accountabi- 
lity. In addition to reviewing each school district's 
contract and fiscal report, MHD examines da,ta f rom 
< the ~SPR in order to judge how effective each local 
program has been. ^ 



Although its historical development has di f fiered front 
the S EA-based Efforts previously ! examined , Oregon's work 
underscores some recurrent issues that characterize all 
efforts at qua'lity evaluation. For example, the use of SPR 
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data to bolster political and financial support for a program 
was a primary purpose in its early years. MHD success in 
Using information for this purpose indicates that LEAs ' desire 
to use data in this way is well founded? efforts expended in 
developing sitch a system can pay off. MHD also found that 
t he ip* system became Useful and was accepted only after 
teacWrs using it were asked to comment on it. SPR's succes- 
sive revisions illustrate both the heed for and the value of 
constantly adapting any evaluation methodology to the changing 
skills and perceptions of the people Using it, as well as to 
changing program goals; 

The differences between Oregon's 'system arid other 
systems are also worth noting for the useful directions they 
can suggest to SEAs. Oregon's focus on individual children's 
progress makes its system particularly strong* In contrast to 
many of the quality assessment efforts' SEAs have developed/ 
the MHD system pays little attention to program inputs or 
characteristics. Instead, it tries' to answer, in the most 

_ y 

direct way possible, the central evaluation question' "What 
is this program accomplishing" and does so by measuring 
change in knowledge and behavior./ Because many SEAs start by 
looking at compliance monitoring, issues , many of their efforts 
to measure quality still ire ly ; hea v i ly on observing program 
characteristics. Oregon's 'approach seems to be a more 
reliable measure of a program's success in meeting educational 
goals. Because oF-*this reliability, Oregon 1 s approach also 
seems to provide a better rationale by which to argue for 
increased political or fiscal; support _ 3^ 

i. * 
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In addition, Oregon's approach also seems to have been 
well-designed to achieve the broader aims of program 
management and accountability. MHD has been able to integrate 
SPR with other agency activities, thereby deriving multiple 
benefits from the initial investment in the system. The use 
of the SPR in developing IEPs is one such example; the 
information SPR generates about student achievement gives a 
new dimension to IEPs , allowing teachers and parents to 
more specific when discussing student goals and accomplish- 
ments. MHD also has used the SPR data to select teachers who 
participated in special in-service training programs and to 
judge the effectiveness of the training program provided by 
Teaching Research, a Division of Oregon's Department of Higher 
Education. In short, these data allow a number of different 
measures of MHD and local school districts 1 effectiveness in 
carrying but their mandate to educate TMR children. 

F . Summary 

A number of summary comments can be made about SE As 1 
efforts to develop and implement program quality evaluation. 
Perhaps most strikingly , the methods Use d by SEAs to measure * 
quality are var ied and represent divergent views about the 
g5als or^quality monitor ing. Figure -I, which summarizes the 
major characteristics of the five examples described in this 
section, confirms that state agencies measure different things 
when they assess program "qua! i ty M . For example, Nebraska and 
Missouri focus- on progrrrm characteris t ics , or inputs. Their 
systems establish program standards (or goals) which are 
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FIGURE I 

Characteristics of State Program Qaai 1 1 y Evaluation Efforts 





SEA Role 


LEA Role 


Basis for Evaluation 


— c 

Products 




• Coordinated develop- 
ment of materials. 

• Convened LEA repre- 
sentatives to review 
materials • 






• Information . for 
local school 
^ districts. 


M lssour i 


• Jointly developed 
evalu tion 
materials with SEA. 

• Will self- 
administer evalua- 
tion methodology. 


• A_ systems analysis 
of educational pro- 
grams, with quest ions 
that address .program 
resources and pol icies . 


Nebraska 


• Coordinated develop- 

m p n t nf mshpr i a 1 <; A n rl 

IIIC II L III u I- vr l. 40A3 QHU 

evaluation standards. 

• Convened advisory 
committee . 

• Will administer 
evalua t ion . 


• Reviewed and 

the proposed 
standards. " 


• Standards and service 

noAl«j wh i rh rief inp the 

U w C2 A. O W|| 1^11 U w I A 1 1 w LUC 

characterise tcs of a 
quality ' program. 


• In format ion for 
local school 
districts .* 










• Information for 
LE As • 

• Information for 
SEA planning and 
budgeting system. 


riui U 1 1 td L Ul L I Id 


• Df* vplnnpfi m^hpr i a 1 q ~ 

W U^. V C A. \_I£JC \J IIIC^LC L I Q X 3 • 

• will administer evalua- 
tion. 

• Will aggregate data 
statewide • 


a T?p \t i p utf rl mAfpri aIq. 

• Have option to com- 
pete for incentive 
grants • 


W C iw^lP 1 ! 1 . y >»0 A O f 3 C V» A 

f i c " ob j e c t i ve s y a hd 
evaluation questions 
which focus on pro- 
gram operations. 

• Numerical standards 
for program accom— 
pi ishmerits • 




• Developed and revised 
materials . 

• Administered moni- 
toring system; 


y. 

• Provided feedbacfc v 
to SEA on effec- 
tiveness of the "\ 
system. 






Cal i fornia 


• General goals and 
specific criteria 
defined as being the 
characteristics of 
a quality program. 


• Written reports for 
LEA and SEA ( com- 
pl iance morii tor i hg 
report) • 


Oregon 


• None . The system is 
supervised by the 
MHD state agency. 


• Administer mate- 
rials to assess 
both program and 
student performance • 


• Increments- of- behavior 
arid/or skill/learning 
change • 


• Report on each • 
student's progress, 
classroom progress 
and district prp-_ 
gress, for LEA and 
SEA. 
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Characteristics of State Program Dual i ty Evaluation Efforts 

(Continuation) 





Other 
Characteristics 


Current State 
of Development 


Relationship to 
Compliance Monitoring 


M issour i 

u 


• Discretion given. to 
LEAs on which pro- 
gram areas to 
assess; 


• To be field-tested 
in 30 school districts 
in school year 198 3- 
84. 


• None. 


Nebraska 


• Use of evaluation is 
voluntary for LEAs at 
this point in time. 


• TP be field-tested 
in four districts 
in school year 1 983- 
84. 


• SEA wants £o coordinate 
but hot comb i he qual i ty 
evaluation with com- 
pliance monitoring. 


North Carolina 


* SEA will award incen- 
tive grants to parti- 
cipating LEAs. 


• Field-tested in 

1982- 83. 

• Implementation in 

scheduled for FY 

1983- 84. 


• l-EAs pilot-testing 
the system are exempt 
from compliance moni- 
tor! ng procedures 

• Eventual integration 
of quality evaluation 
and compliance moni- 
toring • 


California 


• Quality measures are 
interwoven with com- 
pliance measures in 
one system. 


• Used statewide in 
1981-82 and 1983- 
84. 

• Full compliance 

m °ni toring system now 
under revision. 


• Eylly integrated with 
compliance mdnitoring 
system. 


Oregon 


• Materials proved use- 
AMi_prpgram advocacy 
with the state 
legi slatiire . Developed 
by teachers. 


• Used in_Oregon 
since 1969. 


• No relationship to 
SEA compliance 
moni tori ng • 
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believed to be associated with quality; and then assess how 
closely LE As conform to these standards. By contrast, the 
Oregon system looks almost entirely at student outcomes. In 
Oregon's view, program quality should be judged primarily by 
change in students 1 skills arid behavior. This fdclis on pro- 
gram effectiveness means that less attention is devoted in 
Oregon's system to describing local program policies and pro- 
cedures. Between these two are other systems which attempt to 
synthesize the two approaches: these include (1) North 
Carolina's system which blends measures of student achievement 
and program characteristics to measure program quality, and 
(2) California's system which also' measures student 
performance as well as overall program inputs. 

Because none of these systems, other than Oregon's, 
has been tested for ^riy period of time, it is too early to 
determine if documenta t ion of program inputs is sufficient to 
measure program quality. In the absence of such evidence, 
however, the use of student arid program ou' ime measures seems 
to have the greatest potential for substantiating program 
quality. Ultimately, education programs must seek ( and .be 
evaluated by) effectiveness in providing new skills or 
information to a student, and it may be wise for SEAs to 
reflect these goals in their attempts to .design quality 
evaluation procedures. 



varies widely, indicating tha t SEAs will have to choose the 



The SEA role in progr am quality eya 




degree of r e sp on sibilit y they wish to assert in this area . 
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2Q1 of the SEAs discussed here have exerted leadership in 
developing quality evaluation materials and in promoting 
implementation of these systems. In some states, like 
Missouri ^ the SEA role basically stopped there. Much of the 
implementation in Missouri will be left to the discretion of 
LEAs . In other states -, for example f California, the SEA role 
has been much more extensive. The SEA there was involved in 

all aspects, of quality evaluation because it was an integral 

_ _ / 

part of the , state 1 s compliance monitoring system. A position 

midway between these extremes is again demonstrated by North 

Carolina, where the SEA clearly will supervis the development 

and use of the PQE system, but is allowing LEAs to take the 

lead in changing ,i ts content to reflect local conditions arid 

then actually implementing it. 

The consequences of these differing SEA roles are 

likely to be significant. Just developing a program quality 

evaluation system is costly; ' fully implement i ng-~ i t requires 

even more resources'. SEA officials must consider whether they 

can afford this activity without taking resources away from 

other essential functions. For SEAs that want to move ahead 

with program quality evaluation, but do not want it to become 

a statewide activity for which they are then responsible, the 

SEA examples which coordinate system development but do not 

fully implement it may be the most helpful models. 

While the di f f ere rices among SEA evaluation efforts 

are significant, their similarities are even more marked . For 

example, in most of the states discussed here, arid in many 
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other states as well, t he movement toward program qua lity 
evaluation seems to be gaining force . In part, this reflects 
pressures from outside the special education field. The 
national attention being given to the quality of public 
schooling, by blue-ribbon commissions and other research 
studies, cannot fail to have a " ripple effect" for special 
education. However, most of the pressure to examine program 
quality comes from within the special education field, from 
state and local administrators. The SEAs described in this 
chapter have been developing quality evaluation for several 
years; other SEA officials interviewed this report are on 



the verge of designing their own 
monitoring; and , most important jbJ^^yioc a 



ersion of quality 
education officials 



around the country are arguing strongly for ways to document 
the effectiveness of their programs. This Continuing LEA 
advocacy of, quality evaluation may be the we 1 lspr i ng that 
gives this issue high priority for the next several years. As 
long as local special education administrators « must defend 
budgets or compete for new funds, they are likely to want 
methods of proving program success. 

A further similarity among SEAs is that a decision to 
undertake quality measurement means an SEA must be willing to 

reex a rri i iie^ t s own basic _g oa 1 S-^an d - t Sjo s e of local .s^ecj^ajfc 

educat ion programs . Most SEAs find that quality measurement 
sparks a new analysis of fundamental questions such as "What 
is the goal of special education programs?, " "What are our 
expectations for special education students;," and "What are • 
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bur priorities for program staff arid funds?" The SEAs 
examined in this report (as well as the Oregon Mental Health 
Division) answered these questions in different ways, as 
noted , but the process of developing answers seemed productive 
in all instances. An SEA must be willing to invest the tinte 
to sort out these issues, involving a wide range of people in 
the process , before any attempt to monitor program quality can 
succeed . 

Closely re la ted to the preceed i ng issue is the fact 

that t he development of a quality asses sment system is a 

process that involves time arid Usually a wide range of people . 
The SEA experiences described in this section Indicate that 
almost all SEAs go through a process of i 10 clarification of 
goals for e va luation , { 2 K review of alternative evaluation 
systems , (3) development of new materials , (4). building of 
consensus for the new system* (5) training for implementation , 
(5) field testing* and (7) eventual broader implementation. 
Of these activities, those designed to gain user input are 
particularly important because they insure that the system 
design responds to users 1 needs and that new procedures are 
not threaten i ng . 

User participation in developing the system - is 
critical because an ef f ect i ve qua 1 i ty monitoring or evaluat ion 
will change the expectations an d incentives which 
people bring to mpir wor k . Unless a special education 
program is fully effective, a qua 1 i t y mon i tor i ng system will 
alert educators to different priori ties and criteria against 
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which their work will be assessed, " Particularly when 
evaluation standards include numerical measures, as is the 
case in North Carolina's PQE system* tea b;h e r s arid 
administrators adjust their own expectations to those the 
iystem incorporates. These new expectations are u n l\i k e 1 y to 
be viewed positively unless users have had a r^o 1 e in 
develop ing the i r standards . 

For SEAs , men i t or Jun g— eda tea t tonalgaaii t^^raises 
questions about the degree of : control ov- er-lacal programs the 
state agency wishes or is able to exerc ise . One of the main 
differences between the quality evaluation systems here 
described is the extent to which they are mandatory for LEAs . 
For instance, Missouri's plan is completely voluntary? 
Nebraska intends to make the evaluation mandatory statewide, 
but will allow LEAs flexibility in choosing evaluation 
criteria;, Ca 1 i f ornia ' s system makes quality monitoring a 
necessary part of compliance moni tor ing* The SEA decision on 
this issfle is liable to be influenced by factors outside of 
monitoring issues per se : for example, Nebraska has a state 
law requiring a program effectiveness review, while Missouri 
has a strong tradition of local district au tonomy . Even so , 
each SEA must decide whether the qual i ty measurement is to be 
voluntary, used only by fcEAs that find it helpful, or 
mandatory because it is an issue of concern statewide . 

Finally, rtesp LLe^-the stro n g shift towar d qua 1 i t y 
evaluation - ^ compliance m onitoring remains essential . None of 
the SEAs analyzed here believe that quality evalua t ion should 



replace compliance monitoring . They do, however -, share a 
belief that current compliance monitoring procedures miss too 
many dimensions of special education prbgramming. The 
challenge for SEAs in\the coming years will be to pursue both 
goals within feasible\ constraints of staff resources and 
budgets, for both types of program assessment will be 
important, although for different reasons. Compliance moni- 
toring continues to be an essential tool, rooted i n^ the 
federal statute, to ensure that federal and state law^ are 
carried out and to cement the progress that has been made in 
achieving equal educational opportunities for handicapped 
children. Quality evaluation for special education, when 
added to compliance monitoring , can address equally "funda- 
mental concerns about whether programs are having their 
intended effects. 



/ 
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III. SEA MONITORING OF PARALLEL STATE AGENCIES 



Under the general supervision clause of P.L. 94-142, 
SEAs are responsible for ensuring that educational programs 
administered by other human service agencies comply with the 
federal law. For most SEAs , this responsibility represented a 
sweeping new authority. It required the development of not 
only completely new monitoring policies and procedures, but, 
i n some instances, entirely new working relationships with 
other state agencies. 

The scope of an SEA 1 s monitoring authority over other 
state agencies can be very broad, depending upon how a state 
organizes its human services. As the following' examples 
suggest , many different agencies provide care for handicapped 
children in sta te^bpera ted programs. That is: 



• A mental health agency, which may be e i t he r a n 
independent age ncy or part of a larger state health 

— ttepar tmen t , generally operates state institutions 
serving emotionally disturbed children or youth. 
These a re usual ly residential programs providing 
clinical- treatment , edhieat ion , and room and board . 

• A division of mental retardation/developmental 
d isabi 1 i t ies in each state operates programs for 
mentally retarded persons, particularly those who are 
severely and profoundly retarded persons . In mos t 
cases , children and adults are cared for in ^separate 
facilities that include day and residential prog rams ; 
The program of care for children topically involves 
maintenance and education in activities of daily 
1 i ving . 

\ . . _ . * 

• A state department of pub 1 ic welfare, or a state 
children's agency , operates residential f aci lit ies for 
dependent and neglected children. These chi ldren 
include those who have no parents or who have parents 
unable to care for them at home . Care for these 
children may be in foster care or res ideritial 
institutions. While many of these chi ldren attend 
regular public school , others do not . 
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' • A state department _bf corrections administers, cbrreq- 
tibrial facilities for. youth who. are iri trouble with 
the law. Although. these facilities may be called 
"training centers" or "schbqls for boys/girls^" 
their primary mission typically is security arid/br 
rehabilitation. These facil i ties are usually locked 
residential institutions serving youth aged 1 2-21 ; 

Still other^ state-operated programs include children's 

hospitals, skilled nursing facilities for medically frail 

children, and schools for the deaf and blind. Thus, in any 

given state, an SEA may have to monitor the programs of three 

or four other state agencies. 

SEAs 1 attempts, to assert authority over -these agencies 

--many of which had been serving handicapped children during 

those years when they were excluded from public schools — can 

pose political problems. Administrators of stater ope rated 

programs often' resent that* their educational programs now are 

accountable to an outside agency. This resentment surfaced 

repeatedly in the first years of P. L. 94-142 's implementat ion. 

Teaching staff in these f aci 1 i ties (particularly when they did 

not meet the SEA 1 s cer t i f icat ion requ i rements ) were under- 

t ■ 

standabiy unsympathetic to many of the SEA 1 s demands . Human 
service a'gehcy administrators who had to allocate new dollars Q 
to bring their programs- into compliance occas iona 1 ly resisted 
following all of the SEA :, s recommendations. Thus, for most 
SEAs , the task of monitoring th£se other parallel state agency 
programs required a gradual change over time, with progress 
measured , over a period of years as both the SEA and the other 
agencies accommodated themselves to each other's goals and 
methods. 
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SEAs have used a variety of approaches to monitor educa- 
tidrial ^programs administered by paral lei "B-fc-ate agencies. The 
three states' .approach.es that are described here illustrate 
the p^obl6m§ involved in monitoring 3s well as methods which 
can overcome thesp problems.' These approaches include: 

• The .Minnesota Department of Education 1 s mon i tor i ng of 
public welfare arid correctional facilities based on 
written interagency agreements; 

• The Arizona Department of Education's monitoring of 
youth correctional facilities; and 

• The Virginia Department of Education's use of core- 
standards to monitor all state-operated facilities. 

Each of these is described briefly below. 

A m\ The Minnesota Department of Education's Interagency 
Agreements - . 

The most basic approach to monitoring another state 
agency's Bducatibhal programs involves (1) the SEA identifying 

all relevant provisions of P.L. 9 4-14 2 , (2) the^other state 

-J \ / 

agency agre$ i ng to meet these provisions/ and (3) the SEA 

Conducting periodic monitoring visits, to evaluate the degree 

\__ 

of compliance. As is evident in Minnesota , this approach "has 
allowed the SEA to establish its general supervision 
res pons i bi lity f with the parallel agencies — the Minnesota 
Departments bf Welfare and Corrections — consenting to 
conduct their ^educational programs in compliance with state 
and federal law and regulations. Minnesota's experience is 
described in more detai 1 below. 

The Minnesota Department of Ediica t idri entered into 
written agrpemepts with two parallel state agencies 
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in order to ensure that educational programs run by 
those agencies complied with f ederaj. and state law. 
The programs at issue were the Minnesota Learning 
Center, administeredby the be part me nt of Pu bi i c 
Welfare (DPW}' and three correctional facilities 
administered by the Department of Corrections, (DOC) 
which serve youth under age 21 . 

The Department of Education initiated these agree- 
ments, which specified that DPW and DOC would 
conduct thei r programs in accordance with " the State 
Educational Agency Standards relating to facilities » 
staff and supervision," arid follow all applicable 
f ederal regulations . Both agencies also' agreed to 
allow the SEA to monitor their programs and "correct 
d I e f i c iencies cited by t h e .monitoring team. " 
Finally, the agreements specified that the SEA would 
provide in-service training to DPW and DOC staff who 
• provided education to handicapped children. 

The agreements were signed iri_1979_arid 1980. Since 
that time, the SEA has completed one monitoring 
cycle for the DPW arid DOC facilities. This entailed 
two visits, the first, to discuss the criteria for 
monitoring, the second to actually administer these . 
The monitoring successfully identified deficiencies 
that required correction. For example , the DPW 1 s 
Learning Center needed to improve in areas such as 
due process, privacy protection, and use of 
surrogate parents. After a corrective action plan 
was developed, the M LC took steps to cdme into 
compl iance . 

I n the correct ional facilities, the problems were 
moire difficult: a continuum of services was not yet 
available in all facilities. However, the SEA and 
DOC are negbt iating a method for correcting these 
deficiencies, arid SEA officials believe that the 
substantial, progress made by DOC in improving 
programs will continue. (For more info rm at ion on 
Minnesota's policies^ see Appendix E . ) 

Minnesota's monitoring procedures^ based on inter- 
agency agreements, suggest several lessons about monitoring 
parallel state agencies. First, Minnesota's experience indi- 
cates that monitoring", by itself usually is not sufficient to 
improve programs. Tn Minnesota, monitoring was just one part 



of a more comprehens i vr - or king r'»J at ions hip that the SEA and 
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the other agencies es tabl ished ; This on-going relationship 
particularly a general agreement on goals and educational 
methods — has, over time/ brought the DPW and DOC . programs 
into compliance. Conversely, however, Minnesota's experience 
suggests monitoring 1 s key role: it was the monitoring visits 
which identified program deficiencies and led to their cor- 
rection. The leverage supplied by the monitoring process 
acted as a catalyst for change. 

Minnesota's experience also illustrates that 
f o i low-up technical assistance is an integral part of 
monitoring. The Department of Education provided such 
assistance both before and after its actual monitoring. A 
pre-moni tor ing visit enabled DPW and DOC staff to prepare to 
meet the standards which they had adopted in the written 
interagency agreements. Similarly, the post-monitoring 
assistance that the SEA provided to the two agencies, 
including in-service training and intensive work by a 
Department of Education liaison staff member, largely was 
responsible for the speed with which deficiencies were 
cor roc ted . 

Finally, Minnesota's experience indicates that 
monitoring of some state agencies is more difficult than 
monitoring others. The difficulty depends on factors such as 
each state agency's historical ' t ies to the SEA; the resources 
which each has devoted to educating handicapped children; and 
the conducivencss of each state agency's facilities tp 
educational programming. State correctional facilities haA/e 




posed particular problems for SETA monitoring efforts. In 
Minnesota, for example, the DOG programs required more time 
from the SEA than the DPW facility> for all of the reasons 
cited above. Because correctional facilities in most states 
pose these problems , it is useful to look at how another SEA 
handled these difficulties. 

B . The Arizona Department of Educat ion Monitoring of 
Youth Co rrectional Facilities 

Correctional facilities are difficult to monitor for 
several reasons. Historically, they usually allocate few 
dollars for educational programming. Moreover, the education 
programs they have provided often are very basic, offering few 
of the- related services necessary for children with 
handicapping conditions. Most seriously, correctional facili- 
ties usually have not perceived education as central to their 
mission, which is security or, in some facilities, 
rehabilitation. Finally, the characteristics of most 
correctional facilities do not iand x themselves easily to com- 
pliance with certain provisions of P . L . 9 4-142, especially 
the least restrictive environment mandates 

Despite these difficulties, some SEAs have succeeded 
in establishing strong ties with correctional departments. 
Such an example is provided by the Arizona Department of 
Education's development of monitoring procedures for 
handicapped youth in correctional facilities. 

The -Ar izona Department of Ed uca t ion has bee n rnoh i - 
tor i hg the educational programs for handicapped 
children in the State's correctional facilities for 
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several years. this has proven more difficult than 
the Department 1 s monitoring of other state institu- 
tions, but the result has been improvements in the 
nature of education for incarcerated handicapped 
youth . 

Development of a working relationship between the 
Department of Education and the Department of Cor- 
rections (DOC) began shortly after passage of P.L. 
9 4-1*2. At that time , the Department af Education 
( DOE ) hosted a mee ting of all state agencies to 
explain the provisions b f _ the hew federal law. 
Shortly after this meeting DOC requested technical 
assistance from DOE , and a DOE staff person was 
assigned as liaison to the correctional prog rams 
with responsibility for helping DOG improve its pro- 
grams for handicapped children in the four eorrec- 
tional facilities which held youth under age 21. 

A first pre-mon i tor ing visit was conducted in 1980, 
in order to notify DOC staff of the monitoring 
standards. Te chnical ass i stance was provided on 
issues that the SEA's staff knew were weaknesses of 
the correctional program, and DOC .officials indi- 
cated a commitment to sustained improvement of their 
programming. To help DOC accomplish this goal, a 
change in funding was made, whereby DOC became 
eligible for receipt of P.L. 94-142 funds. This 
change was particularly helpful to DOC in giving it 
resources to s t re n g t hen i t s p rocedur e s for 
ide^ t i f y ing and evaluat ing handicapped youth . 



DOE conducted its first full monitoring visit in 
April, 1"9B 2 . " Serious difficulties were ident i- 
fied, particularly on issues of due process, evalua- 
tion arid placemen t , arid IEP development. The SEA 
helped DOC staff prepare a corrective action plan 
which addressed these issues , after which a second 
regular monitoring visit was held in January of 
1983. Several deficiencies still remained, in large 
measure because DOC was still grappling with the 
difficulties of complying with federal policies in 
the context of a secure facility. After this visit, 
DOC staff prepared a corrective action plan , and DOE 
officials believe that this should result in full 
compliance in the near future. 



Theseefforts to improve DOC's programs have been 
very much cooperative ones at the^ staff ^ e y e K'_ 
to initial good working relationships and the policy 
commitments DOC made. DOC has continued to seek 
additional funds for its special education programs . 
In 1982 , DOE gave DOC approximately $ 100 , 0 00 in 
competitively awarded, discretionary funds for 
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program improvements; a similar award of" _Si6'QjO'6Q is 
pending final approval for 1984; In addition, DOC 
wrote a new procedural manual for handicapped 
students, assisted by a special education intern on 
loan from DOE. DOC staff also regularly attend 
training sessions the SEA provides to local dis- 
tricts, which are designed specifically to offer 
feedback on information gathered during monitoring 
visits. 

Arizona 1 s experience demonstrates that steady pro- 
g r. n improvements can be made, even in correctional facili- 
ties, whenever state agencies jointly apply their expert ise 
and resources to the task. Arizona's success in upgrading DOC 
programs grew from both agencies' sustained efforts. These 
involved direct technical assistance, in-service training , and 
targeting of additional funds for DOC programs, as well as, 
-obviously , DOC officials.' -policy ..deals ion to comply with DOE 
standards. Monitoring activities were only one component of 
this more comprehensive attempt to strengthen programs, but — 
as in M i nnesota — the monitoring process established clear 
goals for change, a structure for interagency relationships, 
and a system for periodically identifying remaining deficien- 
cies. Without the pressure supplied by SEA monitoring, 
program improvements may not have materialized, regardless of 
the strength of DOC administrators 1 commitment. 

Arizona's experience illustrates SEAs' special pro- 
blems in securing compliance from correctional programs. The 
issue which caused most of the compliance problems was the 
sheer absence of programs; before the concentrated efforts of 
DOC and DOE, there were no special education programs for 
incarcerated youth. Other difficult issues were the failure 
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to assess all handicapped ybuth> inadequate IEP development , 
and lack of parental involvement. On most of these pbirits> 
Arizona's DOC is now closer to compliance. Moreover, both DOE 
and DOC officials believe that program improvement will con- 
tinue because a process is in place that identifies and 
resolves problems . 

This analysis of both Minnesota's and Arizona's: 
monitoring procedures has f ocussed on how these SEAs gave 
special attention to a specific type of state-operated pro- 
gram. Some SEAs have approached this monitoring task in 
another way , developing a generic approach to monitoring all 
state-operated programs. An example of such an approach is 
provided by the Virginia Department of Education's monitoring 
procedures . 

C . The Virginia Department of Education's Use of Core 
Monitoring Standards 

The Virginia Department of Education, working with 
other state agencies, has developed a system that integrates 
the monitoring requirements of the State Department of 
Education with the licensure and certification requirements of 
other state agencies. 

Four state agencies in Virginia have developed a 
system of "core standards" for Interdepartmental 
Licensure arid Certification, iri art attempt to make 
state examination of public arid private residential 
facilities more efficient. The. four agencies 
participating in the system include the State 
Departments of Education, Corrections, Mental Health 
and Men ta i Retardation, and Soc ia 1 Services . 

The 'impetus for developing the core standards came 
in 1979-80, when certain members of the state 
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leg is la ture received cbmpla i rits f rbffi operators of 
state-operated programs about the duplication which 
occurred when each state agency conducted its own 
ori-site review. _Fbr example* one state agency might 
1 obk just at a facility's, physical coriditibhsL 
agencies might apply dif £ e rent standards of 
personnel certification and staffing; arid each state 
agency could adapt its own standards for program 
compl iance . One legislator prepared draft legisla- 
tion to create a new state agency responsible solely 
for licensure and program cert i f cat ion. To ^Y^^t 
this, the four state agencies proposed an integrated 
review p recess based on "core" -standards, and the 
legislator agreed to postpone .introducing legjLsla- 
tion until this system was developed and tested. 

The four agencies convened representatives from 
pub 1 i c an d p r ivate f a c i 1 it i e s t o assist in 
developing these core standards. These standards 
included the basic licensure and program certifica- 
tion requirements that were applicable to all 
facilities. In addition, program "modules" were 
developed that could be added to the review of 
certain facilities as necessary. For e xample , the. 
education module incorporates many of the issues 
necessary for the compliance monitoring of facili- 
ties that have programs for hand i capped children. 
In add i t ion , an administrative review componen t for 
the state-operated programs specifically addresses 
P.L. 94-1 42 ' s requ i rements . 

The core standards are administered by an inter- 
agency team that visits each facility annually. (If 
a state agency has no licensure involvement with the 
facility, a representative of that agency need not 
participate. ) Usually the team spends three days at 
a facility. The Department of Education's represen- 
tative, then, would participate in both the team 
review and wou Id administer the educati on module to 
assure compliance with federal and state law. When 
deficiencies are noted, the facilities prepare^ a 
corrective action plan which is approved by a state 
agency paper review. Progress then is assessed 
during a subsequent on-site visit. :. 

Initial implement a t ion of the system has been j Udgejd 
a success. The major difficulty has been lack of 
manpower to assure participation of each department 
as appropriate arid some variation in the application 
of standards by different toams visiting different 
facilities, but this is being changed through 
additional training of team members. To date, the, 
iriteVdepartmerital licensure and certification system 
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seems to be satisfying the participating state 
agencies, t h e state legislature, and the 
state-operated arid private facilities. 

Virginians system presents an unusual degree of 
cooperation among state agencies > arid it is noteworthy that 
pressure from certain legislators was in large measure the 
source of this cooperation. While the state agencies pre- 
viously had attempted some rudimentary cooperative efforts on 
their own, it was only when the legislators proposed creating 
a separate new agency for licensure and certification that the 
interdepartmental system was formalized arid launched. 

Virginia Educa t ion De pa r trrie rit officials note that 
the hjw system offers distinct advantages in con eras t to prior 
procedures. As one official put it, whenever SEA staff used 
to review facilities 1 education programs, they occasionally 

C -- .- - - - - - 

had to "step over broken glass" because their responsibility 
did not extend to mon i tor i ng a facility's physical conditions. 
By contrast, each state agency now is able to look at a 
child's total program because of its participation on the 
interdepartmental team. For example, the Department of 
Education team member can assist in reviewing the total 
treatment plan in a mental health facility rather than just 
the program's educational component. Similarly, when 
reviewing a correctional facility, the Depart merit of 
Education's staff member can check a handicapped youth's 
physical environment as well as the available 1 educational 
opportunities. Virginia Education officials believe his 
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attention to the total program adds to a monitoring review's 
substantive value arid efficiency. 

Virginia's system has been constructed so that it 
balances each agency's special heeds for monitoring and 
licensure against the common heeds of all agencies. While any 
one agency would probably add additional criteria' or standards 
to the system, the combination of core standards and add-on 
modules seems to both assure compliance with ail relevant 
federal and state laws and give each state agency sufficient 
data to assess program adequacy, 

Virginia's system also appeals to the need for 
efficiency because state agencies and facilities find it less 
burdensome and less time consuming and expensive. In a 
climate of reduced state resources and deregulation, the 
savings this program anticipates in both dollars and staff 
time make sense. 

D . Summar y 

The difficulties SEAs encounter in monitoring state- 
operated programs are best overcome when th is activity is just 
one part of a more encompassing partnership between the SEA 
and another state agency. Both the Minnesota and Arizona 
policies this section examines illustrate the range of 
activities that can contribute to improvements in state- 
operated programs. These activities' include technical assis- 
tance by SEA staff, in-service training and special funding to 
strengthen educational programs/ as well as vigorous enforce- 
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merit of a corrective action plan. The successful working 
relationship amongst the state agencies in Minnesota arid 
Arizona also owe much to the ^consistency and intensity of the 
SEA >taffs' liaison work. 

Virginia's approach to integrating compliance 
monitoring with the licensure and certification procedures of 
other state agencies represents a different way to ensure the 
adequacy of state-operated programs. Instead of emphas i z ina 
intensive work by the SEA with specific types of facilities* 
Virginia's approach stresses meeting the basic requirements of 
federal arid state education statutes and other federal and 
state human service statutes. This approach views residential 
programming as a whole , rather than segmenting it into its 
component parts. It is an approach which is well-suited to 
state agencies who view their needs as a whole, even though it 
may not provide as intensive a review of educational pro- 
gramming as the more traditional policies pursued by Minnesota 
and Arizona. 
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V; CONCLUSI ON 

While the two topics examined in this report quality 
monitoring and monitoring of parallel state agencies — differ 
substantially from each other, they illustrate in many ways 
the dilemmas SE^s encounter with regard to compliance 
monitoring. Oil the one hand, SEAs have yet to accomplish all 

of the basic monitoring responsibilities mandated by P . L . 

_____ _ _____ _ _ 

94-142. While most states successfully have completed a three 

year monitoring cycle of local educat i«: ss aaoncies — and 

believe that LE As are in substantial compliance with .the 

r 

federal law — effective SEA monitoring of state-operated 
programs still is being developed. This lag is due to several 
factors: the recalcitrance of some institutional programs; 
the political difficulties which SEAs encounter in trying to 
absert authority over other state agencies; and the inherently 
slow pace of change for state institutional programs which had 
developed patterns of care over many years time. For all of 
these reasons, additional progress must be made before the 
general supervision responsibilities assigned to SEAs under 
P.L. 94-142 are fully carried out , and SEA monitoring policies 
for other state agencies reflect this. 

Oh the other hand , SEAs are moving rapidly to advance 
the state-of-the-art of another dimension of monitoring: 
evaluation of program effectiveness, or "quality" monitoring. 
This movement reflects SEAs 1 convictions that. documenting the 
effectiveness of special education programs is essential for 
their future support. The support for such documentation also 
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reflects LF1A officials 1 recognition that local school boards 

are seeking ways to both justify educational expenditures and 

measure the value of educational programs . Thus, at the same 

time that SEAs are struggling to assure that basic educational 

opportunities are avail 'file to handicapped children in 

state-operated programs, they also have taken the leadership 

in developing methodologies for quality monitoring. 

The dilemma in which some/SEAs find themselves is that 

the resources to conduct Monitoring are shrinking. Many SEAs 

ha^ve suffered cutbacks resulting from reduced federal and 

state funding, and SEA monitoring programs have felt the 

•effects. SEAs in all parts of the country have had to 

reexamine and trim their monitoring efforts. In this type of 

fiscal climate, it is difficult for SEAs tc pursue intensive 

monitoring of state-operated programs (some of which remain 

significantly out-of-compl iance with federal law) and 

simultaneously Invest the considerable time and resources 

necessary to produce quality monitoring systems. Many SEAs 

will have to establish priorities between these two tasks and 

decide which direction they most productively can pursue in 

i __ - 

the years ahead. 
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Program ^Effect ivehess Evaluation for 

£ocai^ Education Agencies 
The Nebraska Department of Education 

I . Summary 

The Special Education Branch of the Nebraska Department of 
Education (NDE) has developed , over several years time, a^ system 
for . evaluating the effectiveness of local special education pro- 
grams., Known. as u Program Quality Evaluation (PQE) , the system was 
intended to help LEAs assess their own programs, particularly 
With regard to issues hot covered in the regular state monitoring 
procedures. Under PQE, LEAs have great latitude in choosing the 
-specific evaluation questions they will use, NDE places general 
parameters ^on the evaluation methodology, but LEAs are able to 

I .tailor it to their specific 'heeds. 

I - ' - - ---- - ^ - 

The PQE systems will bar^ield tested in 20 districts in 

academic year 1983-34. NDE will participate with LEAs in 
administrating the PQE field tests, adding this activity to the 
; normal compliance monitoring visits scheduled for the participat- 

ing districts. Evehtually>. NDE plans to implement this system 
statewide . 

II. OBJECTIVES ,/ 

In developing PQE, the Nebraska Department of Education had 
the following goals : * 

(1) To provide a system for LEAs that would enable them to 
document the quality of their programs and thus help 
them represent these programs to local '.school boards, 
parents , and administrators ; - , 




(2) To carry-out a Nebraska state law ( Statute Mo, 43650 > 
which requires that the SEA provide a "detailed de- 
scription of effectiveness of program services"; and 

(3) To initiate a system that eventually could yield state- 
wide data useful to the SEA in on-going planning 
activities and in making policy decisions related to 
state special education programs • 

III. DEVELOPMENT OF THE POLICY 

NDE's interest in quality evaluation is in part a response 
to a state mandate enacted in, 1973 (Nebraska Statute No. 43650 ) 
which requires that NDE provide information to the state legisla- 
ture on the effectiveness of. local; special education programs. In 
considering how best to carry-out this mandate , NDE .officials 
became concerned that data gathered through compliance monitor- 
ing, while essential, did not provide a full picture of a local 
district's programs* They thus began to look for methodologies 
to document the qualitative aspects of local programs as well as 
the procedural dimensions measured by compliance monitoring. 

At the same time, NDE was receiving requests from LEAs for 

V 
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improved methods of documenting the effectiveness of local pro- 
grams. As in many states, special education administrators in 
Nebraska were seeking information that would be useful to them 
when presenting their programs to local school "boards , local 
administrators , parents, and/or advocates. 

To assist them in learning about and developing an evalua- 
tion system, NDE entered into a service agreement with the Mid- 
west Regional Resource Center (MRRC ) to obtain t'rohnical assis- 
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tance in this area. In addition, in the fall of 1981, NDE estab- 
lished a Program Effectiveness Developmental Committee whose 

■> » ... ... . _ . . 

tasks were, first, to select a general program evaluation method- 
ology and then to assist NDE in developing a specific evaluation 
system for use in Nebraska. The Committee \s 20 members iriJludecJ 
LEA staff, higher education personnel, and special and gen ^al 
educators drawn from all geographic areas of the state; 

The Developmental Committee looked at several state systems 
for assessing effectiveness, but soon realized that there was iio 
model that could be simply transferred to Nebraska. Oh the 
recommendation of MRRC staff, the Developmental Committee engaged 
Dr. Robert Brinkerhpf f 6T Western Michigan University * to provide 
additional expertise 1 in evaluation methodology. Dr. Brinkerhof f 
made his materials available to the Committee, providing them 
with a basic model ai a point of departure. The Committee then 
spent a good portion* of its time adapting this general model to 
the specific interests of NDE and the Nebraska LEAs. 

The model which emerged from the Committee established a 

series of service goals in seven major topic areas related to 

special programming. These areas were:* 

• Administration (Relates to the internal operation of 
I the District ) ; 

+ Service Del ivery ( Relates to the identification of 
students and provision of service}; 



*"A Report on the Nebraska Symposium on Program Effectiveness 
Evaluation" prepared by the Special Education Branch of the 
Nebraska Department of Education, p; 16; 
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• Curriculum/Materials/Facilities ; 

• Communication with consumers; 

• Fiscal; and 



• System Aspects (This pertains to the relationship 
between the special education programs and other 
programs or agencies which may be affected by handi- 
capped students). 

The general types of service goals which were idencif ied for each 

area can be seen from the following examples taken from the 



Admi nistr ation topic area: 



1. Board and staff (administration/ faculty, and support) 
roles , responsibilities and authority are clearly de- 
fined and implemented . 

2. Personnel Management --the recruitment, selection; 
appraisal , and development of staff based on program 
goals and needs. This" includes such things as on-going 
supervision and feedback to staff; on-going improvement 
of staff skills and attitudes; and appraisal of staff 
designed to facilitate staff growth. 

3 . Program Evaluation - — evaluation and appropriate modifi- 
cation of the district's program -- e.g. , referral , 
assessment , IEP, services, ultimate impact , and concur- 
rence with regulation and statute. 

4. Staff Communication — positive , accurate exchange of 
information among the staff - e.g., knowledge/ under- 
standing by staff of regular special education programs , 
staff roles , materials* ideas , and finances; utilization 
of positive communication^ skills ; iand utilization of 
conflict resolution skills. 

After these goals were established, NDE convened a symposium 
on Program Effectiveness Evaluation , attended by approximately 70 
educators from around the state. This larger group (1) reviewed 
and revised the standards; (2) identified the types of informa- 
tion that an LEA would need in order to evaluate each of the 
service goals; and (3) identified potential pitfalls within each 



service goal . 
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The revised goals which emerged from this symposium are not 
intended to be followed verbatim by the LEAs that will eventual- 
ly use the system; NDE sees its role as putting constraints or 
parameters on the LEAs 1 use of the model . That is , NDE will help 
LEAs determine (1) how the LEA should L lentify questions related 
to each service goal r (2) who these questions should be directed 
to, and (3) how the LEA will collect the necessary data. Within 
these general guidelines, however, the LEA wil J determine the 
specific questions to be used in its quality ev3 1 dnt !:6n ; 

I V . IMPLEMENTATION 

NDE will be field testing the Program Effectiveness Evalua- 
tion model in 20 districts in the 1983-84 academic year . Use of 
the model will be combined with the monitoring visits that NDE 
staff would be conducting" in any case in these districts (either 
maintenance moni tor ing visi ts or full cornpl iance monitoring 
visits). NDE staff estimate that, in most districts* compliance 
monitoring will be performed during the first day of the visit; 
then the remaining 1-4 days will be used for testing the evalua- 
tion model. 

At the conclusion of the first evaluation year, LEAs will 
submit an evaluation report to the state. Those LEAs partici- 
pating in the field test will not have to submit a full correc- 
tive action plan as the result of the compliance monitoring 
visit. Instead, NDE staff want to see if some of these same 
issues can be addressed through the evaluar. ion plan. 




V. ANT I "I PATE D EFFECTS 

In the words of ari NDE official/ the Program Effectiveness 
Evaluation system is designed to meet "the heeds of the LEfts 
firsts and the SEA second . " Specifically, the system is expected 
to (1) assist LEAs in representing special education programs to 
local boards and commissions, and (2) identify areas in which 
improvements in quality can be made. NDE does expect to reap* 
benefits from use of the system as well. First/ the system 
should meet the requirements of the state statute / referred to 
above. Second/ Program Effectiveness Evaluation will give NDE 
staff a new type of information <vr. special education programs 
statewide/ which should be useful in setting priorities and 
future NDE goals. 

For more information on Nebraska's system contact: 

Mr. Don Anderson 
Compliance Direc tor 
Special Education 
Box 94937 

301 Centennial Mall .South 

Lincoln/ Nebraska 63509-4987 

Y 
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AN EVALUATION MODEL FOR' LOCAL SPECIAL EDUCATION PROGRAMS 
THE MISSOURI DEPARTMENT OF ELEMENTARY AND SECONDARY EDUCATION 




AM EVALUATION MODEL FOR LOCAL SPECIAL EDUCATION PROGRAMS 

Missouri 

<■■ 

t 

I. SUMMARY 

The Missouri Department of Elementary and Secondary Educa- 
tion is developing an evaluation model to be used by , LEAs in 
self-assessment of their special education programs, The SEE 
model (Special Education Evaluation model) has been put together 
by the - SEA , working closely with representatives from local 
districts. It incorporates a series of questions that address 
the effectiveness of local programs on six dimensions: identifi- 
cation of handicapped children; evaluation/diagnosis or revalua- 
tion; IEP development; placement; IEP implementation; and annual 
program review. ~ 

The SEE system will be pilot tested in school year 1983-84, 
out Missouri has no plans to make the system mandatory for all 
LF,As. Instead , the system is intended to be used by LEAs that 
find it useful to document the effectiveness of their programs 
with school boards* parents* and local administrators. 

II. OBJECTIVES 



In developing the SEE model , the Departme^*: of Elementary 
and Secondary Education (DESE) had several objectives: 

(i) First , DESE 1 s overall goal was to improve local special 



education programs and services. By identifying 

A- — - 

program strengths and weaknesses, SEE should help ad- 
ministrators to make necessary program changes. 
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(2) Second , the system was intended to help LEAs justify 
their programs arid funding. By documenting the quality 
of their programs in a systematic fashion, LEA adminis- 
trators are in a better posi tion to D defend current 
funding levels or request support for program improve- 
ments. 

(3) Finally, although the system is not mandatory, DESE 
hopes to use the feedback it provides to identify 
training and technical assistance needs throughout the 
stake. 

TIT. DEVELOPMENT O F THE POLICY 

DESE activity in developing the SEE model grew from a number 
of sources. For several years, DESE special education staff had 
been interested in quality evaluation, particularly as the result 
of some work on student outcomes that been done within the state. 
Observing this work , DESE staff became convinced that outcome 
data alone did riot provide all the information needed by local 
and state administrators. They felt that a full evaluation of 
quality also had to address program effectiveness and to estab- 
lish a relationship between the character is t ics of programs and 
the outcomes of programs. 

The interest in quality evaluation among DESE staff was 
reenforced by LEA requests for technical assistance in developing 
a quality evaluation system. LCAs wanted to be able to measure # 
their effectiveness on dimensions, that went beyond those included 
in the state 1 s compliance monitoring system. LEA admins tra tors 
believed it was particularly important to be able to document 



program effectiveness when representing special education to 
local school boards. 

As a result of this strong LEA interest in quality evalua- 
tion, DESE staff began looking at evaluation methodologies that 
could be used in Missouri. They were particularly impressed with 
the materials developed by Dr. Robert Brinkerhof f at Western 
Michigan University, and after hearing Dr. Brinkerhof f at an 
evaluation meeting convened by the Nebraska Department of Educa- 
tion, contracted with him to assist in the development of an 
evaluation model for Missouri. 

To ensure that the evaluation model reflected LEA interests 

i 

and concerns, DESE convened a developmental group of approximately 
20 local representatives. Beginning in February, 1283, this 
group met each month to develop program standards and questions 
within the general structure provided by Dr. Brinkerhof f 1 s meth- 
odology. Back-up staff work for the group was provided 1 by staff 
from the Midwest Regional Res-rrce Center ( MRRC ) , who had 
been working with DESE for some time around the issue of quality 
evaluation.. 0 I 

The SEE model which the group developed uses a ; system* 
approach to examine special education programs. It identifies 
six major components required of all special education programs 
(see Figure I, taken from a summary of the SEE system) and 
then poses questions to determine how well each of these compon- 
ents is functioning. For example, questions related to the third 
component, " IEP Development" , include : 



1, Do annual goals reflect student needs as specified in 
the present level of performance? 

2, Are ail needs as specified in the present level of 
performance listed in annual goal statements? lihy or 
why not?) 

3, Are there annual goals that are unrelated to the pre- 
sent level of performance? 

In each of the program areas , the SEE model will try to look 
at (1) resources, ( 2 ) services , ( 3 ) outcomes , arid (4 ) feedback , 
and thus achieve a view of the program as it functions as a 
whole. ( In the systems approach used by,. SEE , these aspects of 
program functioning correspond to (i) input, (2) process, (3) 
output and (4) feedback*) Thus, in its questions, the SEE model 
incorporates two dimensions : the programmatic elements shown in 
Figure I , and the syscemic elements listed above, 

I V . IMPL EMEN TATION 

DESE plans to field test the SEE system in the coming school 
year (1983-34 ) in 30 volunteer LEAs. These 30 are expected to * 
consist of approximately 50% that were represented on the devel- 
opmental task fierce and 50% that were not. The initial evalua- 
tions may take from i/2 to 2 years , depend i rig on the in tens i ty 
and degree of follow-up in each LEA. 

Prior to the field test, more training will be provided by 
Dr. Brinkerhof f to DESE consultants and to the task force members 
in tire expectation that they will in turn be able to train other 
LEA staff using, the system. The possibility of providing a small 
amount of funds to the LEAs involved, in the field test has been 



4 



Figure 1: 

Special Education Services Flow Chart 
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considered by DESE* but it may hot materialize because of budget 
constraints. 

The SEE system ' wil be implemented separately from the 
state's compliance monitoring system, although DESE hopes to ase 
the evaluation data as a complement to the monitoring data. * 

V. EXPECTED EFFECTS 

; DESE has deliberately designed the SEE system §s one which 
depends on the interest and' activity of LE^s. DESE staff be- 
lieve that "evaluations of. programs and service are conducted 
best by local personnel. 11 * Thus, DESE has no plans to mandate 
the system statewide. 

However, based on the ££ft re ; : : to the system, DESE 
believes that it will have, a number on aft'edts including 

( Assisting LEAs defend and promote special education 
programs ^at the' local level; s 
\ (2) Giving feedback to parents in local programs.; and 

(3) Assisting DESE staff - identi fy technical assistance and 

f - * ----- - -- 

\ training needs! at least among / the LEAs using the data 

and potentially on a statewide basis. < p 

Most importantly, DESE bel ieves that evaluation should lead 
to change . In the long 'run, information generated by the SEE 
system should allow education to measure progress against/ goals 

I i 

and take action to improve services for handicapped children. 



* "Evaluation Preview Special Education Evaluation Model for • 
Missouri" , draft materials for Department of Elementary and 
Secondary Education , page 1. ' v 



// 0 

For information ori the SEE program contact; 

Mr; Ted Hi eke 11 . / 
State of Missouri" 
Department of Elementary 

and Secondary Education 
P.Oi Box 480 

Jefferson Gitv, Missouri 65162 
(314) 751-2965 
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A SYSTEM' FOR PROGRAM QUALITY EVALUATION 
THE NORTH CAROL IMA DEPARTMENT OP PUBLIC INSTRUCTION 
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A SYSTEM FOR PROGRAM QUALITY EVALUATION 
NORTH CAROLINA 

I . SUMMARY 

The North Carolina Department of Public Instruction, the 
Division for Exceptional Children (D:^)> ha; developed a system 
to measure the qua! i ty and effectiveness of special education 
programs for handicapped children throughout che state. This 
system, "Program Quality Evaluation" (PQE ) , j developed 

expressly for North Carolina, using input from the eight regional 
coordinators and other key DEC staff. Revisions to PQE were made 
after it was field-tested in two school systems during the 1982- 
33 school year. Sixteen more systems (selected from among those 
that responded to a stat -issued RFP ) will serve as pilot sites 
for PQE this fall. Eventually, DEC plans to combine these eval- 
uation activities with North Carolina's ongoing compliance moni- 
toring procedures . 

11 • C^ECTIVES 

The Program Quality Evaluation system is designed to provide 
LEAs across the state with a mechanism for determining program 
quality. Through PQE, it will be possible "*co ascertain the 
existence of quality in special education programs, determine 
educational gains of student learners in the programs, and estab- 
lish specific needs for technical assistance from the state 
•:<iiK:a h ion /woncy . " * Further, it is anticipated that data which 



* program i 4. va 1 u a t ion Manual, Preface by Theodore R. 
Brain, Director, Division for Exceptional Children, p. ii 



So 



the LEAs collect using PQE will not only b^jxsef ul to them when 
justifying expenditures to local; school boards, but also will be 
useful to the SEA when conferring with the State Board of Educa- 
tion and the General Assembly to report on the eff ec t i veness of 
special education programs statewide . 

III. DEVELOPMENT OF _THE_ HLI CY 

In 1980, in response to LEA requests for technical assis- 
tance on the topic of quality evaluation, DEC contacted other 
state education agencies to identify evaluation models that could 
be adapted for use in north Carolina. Because this area was new 
to almost all SEAs, r discovered that there wr -? few operating 
models and decided t 1> lop its own. 

In the f -11 of 19'6±, a DEC staff member ati.^ndec a Title VI -D 
meeting in Massachusetts and became familiar with evaluation 
designs developed by Associates in Professional Technologies, 
inc. (APT)-, a small consulting firm in- -Connecticut. An APT 
representative subsequently met with DEC staff in North Carolina 
to discuss evaluation systems that. could be responsive to the 
needs expressed by I As. APT prepared a proposal for system 
development which was submitted to the North Carolina State Board 
of Education and approved for a contract period from August: -, 1982 
to Juno , 1983. APT took on three responsibilities tor DEC: ( 1 ) 
the design of the Program Quality Evaluation Model and production 
of 500 PQE manuals; (2) the' training of key DEC staff in the 
evaluation process to enable them to provide technical assistance 
as -neodLd to LKAs; and Uj the modification of the model/inanual 
based on pro- and post-test data analysis from two field-tests 
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which were conducted in Pitt arid Union County school systems. 
Throughout the system development process, APT and DEC staffs 
through ah Advisory Committee* worked closely together . The total 
cost of APT 1 s development of PQE -, a system not previously used in 
any ether SEA, was 543,008. 

IV; IMPLEMENTATION 



PQE is a goal-based program and includes goals in three main 
areas: Determining Learner Gains/Outcomes ; Locating and Evalu- 
ating Learners ; and Placing Learners Appropriately. For each of 
these areas, PQE establ ishes program objectives and specific 
evaluation questions. For example, part of the sequencer within 
the first gc 1 area is as follows: 



Goal 1: Students will make satisfactory progress in the 
specific curriculum areas in which they are enrolled arid 
dev.: lop a positive self-image for learning and work. 

Objective 1.1 : Studen ;s successfully demonstrate compe- 
tence in appropriate curriculum areas at levels commen- 
sura t<". vi th abil i ty arid 'hand i capping cond i tibn . 

h valuation Question 1.1.1: Are reading competencies 
attained _ comr ns urate with students 1 abi 1 i ties and 
hand i capping cond i tions? 

- Vocabulary 

- Comprehension 

Cvalua t j on Question 1.1.4: Are vocational coinpe te n- 
cies attained commensurate with students 1 abilities 
and hand i.eapping cond i tions? 

For each of the specific evaluation questions, LEAs are expected 

to establish a numerical standard to mnasure satisfactory program 

performance. For example, for Evaluation Question 1. 1,1 above, 

in LHA ryji: id decide th . 0 1 of students in a program should 
attain the desired reading c ji.lpo tone i 053, Kov "valuation Question 
' above, the nunorical standard omld be G 0 . In btfior 



words, the stah iairds sot by each LEA can vary from question to 
question/ and different LEAs will set their standards at differ- 
ent levels > reflecting varying expectations of local programs. 
Eventual I y , DtZC may assist LEAs in setting these numerical stan- 
dards using data obtained statewide to ensure that LEAs do hot 
set expectation levels too lew. 

DEC .'ill be implementing the Program Quality Evaluation 
Model on a pilot basis In 16 (out of 142) LEAs during the 1983-84 
schc 1 ; \»c; V;.esG L F. .\ s will be competitively selected; DEC has 
issued « .\\.i:'ci; : ; \ : for a PQE Incentive Grant Program which will 
provide f'.iriUs to assist with the implementation of th< self-study 
evalu'itic. . Grants will be awarded in amounts from $2,000 to 
$10,000 based on the December .1 , 19 8 2, handicapped headcount in. 
the LEA. Applications will be reviewed by an : ncentive Gran- 
Review Committee (composed of BRC staff) and notification :>f 
funding for the sixteen selected sites will be made following 
State Hoard, of Education approval in October, 1983. 

An initial meeting of the local d irec tors from the -sixtr n 
pi Lot sites will be conducted in November of Liu's year, during 
t vi 35th Annual Conference on i-Jxcejpt idrial Children. During 
November and December the DkC Regional Coordinators will conduct 
l n-rerv ice training for the local personnel , and the sixteen 
pilrt i/KAh will implement the Program Quality EvaJiiat model 
from .January to Apri 1 , 1984. 

Following im; 1 -mentation/ tcli 'M; the sixt- n pilot LEA;s 
will submit a n i rvicjemtMi t >l,vi to bhC to report the Ci-uiings of 
Mtuir evai.Kit. n and to describe >ny corrective actions they 



believe necessary. The management plan will specify timelines to 
correct deficiencies arid, address areas of high priority. A 
meeting is planned for Juno, 1984 allow local directors from 

..the si;;ce>. ^ sites to share experiences and offer recommen- 

dations for m'ther PQE implenentation. 

V. EFFECTS. 

It is anticipated that, through their participation in the 
PQE process, LEAs will have qualitative data for use with 
their local school boards to help gain progre r n support and to 
acvriire ~oeded funds. In addition, LEAs using PQE will be able 
to note program deficiencies more readily and to correct them. 

Vrie SEA 7 by examining LEA mana^ment plans, will be able to 
note state-wide trends which should prove useful in determining 
SEA sti'.f training needs and in planning LEA in-service training. 
The data obtained by DEC should also allow eventual' evaluation of 
state expenditures in terms of prograrr outcomes. 

HEC Iready engaged in some nationwide dissemination 

activiti* ^s planning others. It has distributed a copy of 

its l y Qc, manual to every SEA, each RRC , and riany national organi- 
zations, as well as to other interested parties upon Request. In 

F*?hz .j . j r l b?o4, DEC; si <iN: v/iit conduct a Lwo-Uciy workshop <^h FQE 

U J 

at the 1 C oai Conference on Program Evaluation to be hold in 

Tn:'i;;'i , ' lurj. a; 

K'->r more information, contact: 

The Division for Exceptions.- Children 

North Carolina Departmon t c r - Publ ic Instruction 

Ra igh , north Carolina 27G02 

(91. j 73J- 3921 
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PROGRAM REVIFW 



THE CALIFORNIA DE PARTMEN L' 01- EDUCATION 



So 



California's Program Review system is described on pages 
1-11 of this chapter. To better illustrate the nature o: the 
system, sample materials from Californi 's Program Reviev; 
Handbook are attached. For additional- information on 
California's system, contact: 



Mr; Jack Hazeohmp 

California Department of Education 
; >ch Floor, 721 Capitol Mall 
Sacramento, California 

( 916 ) 323-47^ 



paut i: b.-kRVfkw of •niK sklpa program 



Purpose of Program Review 

The hr ma ry purpose of program review* whether it is an iht rn:i ' review or 
one cond.. ' » ; d by an outside agency, is to improve the effec of ^.is ^.ction on 
the sti.' Jt is a neans for developing and sustaining a high-q t. ai: i fy 

edueat: -.1 ;»• -.ram for each student. 

Program review can be an integral part in the program improvement cycle of 
planning, translating the planned program into action, evaluating the degree of 
implementation of the program, and modifying the program record ingly. Program 
/eview findings can be of value in discussions and decision making about the 
design and implementation of programs- In short, the program review process 
yields i n i o r.na t ion that is essential to effective program dove iopment— informa- 
tion about what is working weil and why and what should be changed a " how. 

Uses of Thi s Handbook 



The program review process described in this ! . . book is designed to 
complement, not replace^ the existing on go ; ig planning an^ evaluation processes 
in i school . S E L PA , or agency. Reviewers should make use of all evaluative 
information developed within the SELPA/a^ency and use this handbook as a 
fraiewort for organizing the information. 

Reviews conducted by reviewers from outside the SELPA can provide a fresh 
viewpoint and independent validation of internal review findings. Toward this 
end , the program review conducted for the St'te Department of Ed'^atioh by a 
^ :<;te review team will be based on this hanr' nook. A school * V 1ELPA, < '■' agency 
can use ail r part of this hand Jo ok over a longer period of time to evaluate 
its o* f n prbijra.;. Such use of this handbook by the people at a school, SELPA, 
M r agency prid^ to a state-conducted review can be extremely effective in 
helping to fmprrve the local program. Familiarity with the review criteria and 
procedures in this handbook will help staff and parents communicate effectively 
wi th reviewers and will h :ip reviewers obtain the data they need to make informed 
jud ;:;;:jent s ♦ 

Process or Program Review 

s " 

This r;i-r t ion describes seq iieii it in 1 ly how _a prog r.i h rev i ew is conducted by a 
State \j\ ' i>.i i* i \ie at of Education review t.c.un. While the description and instruc- 
tions arc ;irec:» 4 d primarily £-wjird reviewers who r<j members of a state— 
rdndcc;ted review learn, the .school , SELPA, or a gen y may adopt, as appropr in te 
parts of tiiis process for use in self-review. Appendix A contains three 
checklists o. responsibilities for team members, team leaders ; and the SELPA/ 
agency which ;;ive more detailed information about the process; 



kov: »w of the Local Pi m 



the local plan for special ?duc~tion is the first contact the reviewer ha?; 
with the procedures:; policies, and programs implemented in the SBLPA/agency; 
The team leader will review the plan for a thor^: : ,h understanding of the BELT A/ 
agency's goals and implementation proce "ses, priar to interviews in the SELPA/ 
anciicy or school site visits. The contents of the plan viy. theiri be discussed 
wit. the review teaia. 

Initial Team Meeting 

The team of loviewer:; jtH meet together, usually the night before the 
review begins, to: 

o Discuss the local plan. 

o Dip curs S : X?A,' agency and school information presented by the team leader. 

o F % . ■ • i ew ma j o r i i nd i rig s of previous reviews. 

o Plan strategies for the review.. 

o Define reviewer rol^s and responsibilities. 

o Designate meeting times during^the review to "touch base" and share 
p ~ rcepti6ns , and develop SELPA/agency final report. ; \ 

s 

u » " : f.w the scheduled* visits, 
o Lo know each other's interests and strengths. 



Collect ing Information 



In finding out about the quality of the school and SELPA/agency proc.edineJ , 
the reviewers use. thre' asic review methods. They are (1) classroom observa- 
ticn; (2) interviews- _Jtn formal and informal; and (3) the review of records 
arid documents. I'dch cf chese methods is used in conjunction with the others to 
verify information and iindings. For example, by talking with the teacher and 
students about the observed activities, the review* r can ^erify the accuracy and 
completeness of his or her observations. 

/»■ 

* 1 1< 1 ; t t o >(> m 0 b * j ■ * r v i * i. p 

Through classroom obse rva r * on f the reviewer gathers basic information about 
the school program. The reviewer visit'; the 'rM^fiar ci-issro..; n s, special day 
classrooms, resource specialist p* grains f and ii M m r.; M< or areas where e ; iica- 
t i oil and related service:; are rii ided h iiuliiKui il^ wi ; n" exc?7viohai no oris. 

Classroom observations ir-citid** ! i" ^ ; i.i * in; e r /lews :\th tudeiil and s ta f C . 
based upon what lias b ^ observed. : 



Interviews 

Interview. 4 ; are an extreme 1 y important socirce of information about the 
program. The basic information gained through classroom observation and review 
of records and documents is verified, clarified, and expanded through the 
interview process. Interviews enable the reviewers to learn how the program 
was developed, how it is being implemented t arid how it could be improved.. 
Students, parents * .teachers * resource specialists^ program specialists^ 
.idriiriistrr. ".brsj arid other servi e providers are interviewed. 



Interviews serve the following major purposes: 

o To verify data obtained from other sources 

a Tcj collect data not yet ga the red f ron other sources 

o To resolve discrepancies in data collected 

o To give people an opportunity to share past experiences , present 
realities, or future plans 



o To v»i ve people an opportunity to ask questions 

o To ge;-rate ideas about how the program could be improved - 

Review of Records arid Documents 

Information gathered through classroom observations and interviews is 
verified* ex pan led, and clarified through the review of such records p.-: 

o Kofei ral notices, assessment plans, and assessment data 

o Individualized education programs (lEPs ) 

Minutes of school site counci 1 /schoo 1 advisory comni t tee /community 
advisory = • oni" »i ttee meetings 



a Contracts 01 agreements with other r»ericies 

o " Mva uati -ns of star f de.elopment and ,>arenc education ict ; /ities 
o be*: son plans and ther.-' py records 

o .Wvvs i er t ers and other communications to parents .ind the community 
o i'r ri?dur:" ilri'' policy hand b< ;fcs 

VI • :n;n evaluation Irif brrn.i t ibri ^ including se) f-ri^ i; i'.: firidii.^i 



3 

B2 



Parent and : Conmun i t y Input Meeting 



A parent ^itid coitiiivinity input meeting is scheduled for one afternoon rr 
l' v e ii f 1 firing tho revie Often this meeting is _schei±iiled in conjunction with 

.-..eLLiny of the cbinniurii' < advisory committee (CAC) to provide its members an bppbr 
Lunitv to discuss activii :-.*s and goals of the CAC. During this time members 
community will have ah rturiity to voice their opinions regarding any aspect 

of the s ; rial educat? ram. This inforinat ion will be used in making 

de r e : ,:it na z ions about ; , m's strengths and needs, as well as suggestions 

for mprovement . 

Ident ! i'c'.i: : oil • ■: Effective Programs and Practices 

Throughout- the / . • . , Is team members lea rri about effective practices and 
programs, they will complete an abstract describing that practice or program, 
lis iri^j the form in Appendix D. These abstracts will be compiled each year by 
the ualiry program uriit and made available to local education agencies. 

Final School Report 

vleno rally, £two reviewers will spend c wo' days collecting da* at each Li rgc 

school site (less time at the smaller school sites/)- A*, the co. : iisi on of the 

data collection at each site, the team members will deliver an informal oral 
\r — ■ _ * 

'r port to the school re fl :.ti ..\ whether or not standards have been met, areas 
b:X^t rerigtlis and needs t ind icat ions of noncompliance, suggestions for improvement , 
and suggestions tor refinement. 



Ft na 1 Team Meet tng t 

All LP n.crmbe *-s wi i 1. meet together,' usually the evening before and the 
lart clay of l'\e 'eview, to wrap up the final oral report to the SELPA/agericv; 
At this time team menbe rs agree upon whether or hot the stand rds have been 
met : , areas of strengths and heeds^ «*ind findings of ^oncompl "i ance _They also 
develop suggestions for improving a program which has riot; met th<# s'tariHa rds 
and suggestions for re fin: > practices or programs which have met the 
standards. 



Kinai SELPft/ Agency Report 

On the J ant day o 4 i :hv r> v. ?w, team in it.be rs deliver the final c al e po r 
f i rs mi f o: 1 i y to t h e^: S E L FA ad m i n i s .: ra 1 1 o ri and sec ond , f o rma 1 1 y , to t h e 
SELPA, district, school staffs, a ind community. The same findings of . h r: 
'."eviv-w tea 7u ii r(j incorporated " nto a f T r^il w r i 1 1. < • " report arm .^cPit briers. ♦ e n.^e 
SLLPA within 20 working days of cohijil e L i bii of t l v' rev i ew. The final report s 
iie. 'ie r i hi.-d in more detail Liter in this Handbook. 



PART ill: ORCANf NATION OF THE I IAN i) BO OK 



This chapter Explains the format of the Program i.eview Handbook. This year 
in the program 'review we are interested in four main areas: 



1. 
2. 

3. 
4. 



; Referral /assessment 
individual] zed education program ( 1 EP) development 
Placement and iEP implementation 
Administration ' 



The first three areas will be used at the school level , arid the last area 
will be used at the S EL P A/agency level. For each area* criteria (standards) 
have been developed, reflecting what is expected to exist in any program. T 
criteria include standards taken directly from federal or state laws and reg 
tions, as well as standards representing basic co-nmon sense or good practice 
Each school /SEEP A /agency r ^ measured a^airtst the criteria to determine sere.: 
and needs of the program, and areas of noncpmpli ance . The criteria are wr ' 
-in the for i of paragraphs , which appear in the center of .pages 7, 13, 18, f 



• ;hs 
:en 
: 24. 



Each set of criteria in structured around iber "organizers,' 
consistent throughout all criteria. The drga riize irs * are : 



wh i ch a : 



1. Procedures/document ar i 

2. Staff involvement 

3. Parent Is tudent inv: 1 
A. Staff development 



n 



■ont 



-in each 



Each organizer is a separate p 

ideas" are capitalized;.. For exarupx ■ > in the first paragr i 
for "Referral /Assessment , " you will see the words "ONGOING PROCESS TO V 
alL in capital letters, re fie mg the first key idea within the organ! 
"procedures/documentat ion. M 



organizer, the "key 

of rjie criteria 
IND" 
rganizer 



?n rhe criteria for "Adminis trati 



ideas. 



•/ 



Mic organizers the key.. 



Fol lowing each criterion ar^ 'Areas and Sources." These are questions 
developed !o help the revivwet collect data. To th? right ;vf each question is a 
gviJ i -/Ring where t! : reviewer should no to find the answer to the question. 
Sources include; 0 ~ Observation, D - Documeata tiori^ R ~ Regula r teachers , 
instructional :ides^ s ite principal* special program teachers (e.g.. , ^Chapte'r 1, 
Mil'l* i — IJ'i i Hi! J - S — Spec l_u education teachers , a lues , DIS, F -* Parents vanu 
stu cats wiih ^eeo t ionTl^ needs as; appropriate) , A - SE LP A/age icy 1 minis t ra- 
tion, I - Iiicei view administration, Ph - Pupil records. An,; observ.i i ion guide, 
record revf cw .^uide, and interview guides for regular and s pecial education 
sta/'f; parents, students and administrators containing % q'ues Gions as indicated 
; r. rhe grid wi:* t be used by reviewers to collect rm or ira t iryn;* 



Qiu.^tfons are numbered consistently in the guides for 
first digit indicates the i:rite rion or chapter numbe r : 



-rosis -re f e rer.c i ng. The 



9-i 
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1 = Ref erral/nssessment 

2 = IKiP development . " ' " ( 

3 = Placement and IEP implementation 

4 = Administration ' 

The second digit indicates the number of the main questibri (taken directly from 
each of the ^criteria). The third and fourth digits indicate sub-questions 1 
necessa.y to answer the main question." The number in parentheses after Jthe ^ * 
question is a cross reference to related questions in -other criteria chapters. 

* » 
■ The fourth part of the handbook, includes the format- for the finjil report, a 
sample final report, and a description of both the f-inal school repo : a> * v *^ 
: inal SEL-PA report.' ' , 



i*\KT 111: CRITERIA FOR ASSESSING" COMPLIANCE AND 



QUALITY OF THE SPECIAL F. DUCAT I ON LOCAL PLAN AREA 



CHAPTER I : REFERRAL/ ASSESSMENT 

There is an ONGOING PROCESS TO FIND unserved and 
inadequately served children , birth through twenty- 
one years o!f age. Attempts have been made, to 
MODIFY THE (REGULAR EDUCATION PROGRAM before 
referring t\\e student for special education. f 
REFERRALS A&E PROCESSED FOR ALL potential special 
educa tion s tudents within the required time lines. 
Assessment materials and procedures are NONDIS- 
CRIMINATORY. The assessment team includes PER- 
SONNEL APPROPRIATELY TRAINED to administer and 
interpret test results. Each child is ASSESSED . .. 
♦MEN REQUESTED by the child's teacher' or parenu.-^'" * 
INDEPENDENT ASSESSMENTS are considered by the IEP 
team. An ASSESSMENT PLAN is developed as a result, 
cf refertal. WRITTEN PARENTAL CONSENT is obtained 
;>rior to assessment. The student^ is ASSESSED IN 
ALL AREAS related to th~ suspected disability.' NO 
o INGLE PROCEDURE is the sole criterion for deter- 
mining olaceraent of the individual-/ ASSESSMENT is' 
COMPLETED and a REPORT IS WRITTEN before the if?" 
meeting. 

Assessment is ronducted by a MULTI DISCIPLINARY 
TEAM, including a specialist in tike area of 
suspected disability. Most STAFF UNDERSTAND AND 
USE the ref erral/assessi it process. Assessment 
RESULTS ARE SHARED with IEP team at or before the 
meeting. : 

'PARENTS AND the COMMUNITY are informed that they 
CAN MAKE REFERRALS. PARENTS. ARE NOTIFIED that 
their child has been referred for assessment arid 
possible special education services . Parents are 
informed of all DUE PROCESS JUGHTS. 
i 

STAFF DEVELOPMENT AND PARE: 'T/COMMUNITY EDUCATION 
BASED ON NEEDS ASSESSMENTS have been provided in 
che area of referrai/assessmenL. 

t I 



i . 0 REFFJ^RAt/ ASSESSMENT 



Areas and Sources 



Procedures 



/Docuji 



ehtatibh 



1.1.0 What is the ONGOING PROCESS TO FIND unserved and 

inadequately served children, birth through twenty-one 
years of age? [E.C. 56300, 34 CFR 300.220] (4. 1.4) 

1.1.1 Tell me about the process to find and refer students 
who might need special education. fE.C. 56300; 
E.G. 56301; E.G. 56302; 34 CFR 300.220] (4.1.5) 

1.1.2 Do you understand how to use the referral process? 



1.2,0 What attempts have been made to MODIFY THE REGULAR 
EDUCATION PROGRAM before, referring the student for 
special education? [E.C. 56303] 



1.2.1 



What kinds of strategies do. you use to help the 
child before you refer? Do you get assistance 
from the special education personnel in developing 
strategies? (4.1.5.1; 4.1.5.2) 



1.2.2, Are programs such as Chapter 1 (Title I) utilized 
before referral? (4.1.5.8) 

1. 2. 3 How do you assis t the regular education teacher 
before the referral? 

1.2.4 What kinds of tilings did the school do to give your 
child extra help before he or she was referred? 
[E.C. 56303] (4.1.5.1) 

1.2.5 Did the school talk to you about your child^ 
problem before referral? 



1.2.6 



Do records verify that efforts were made to modify 
the regular program before referral? [E.C. 56303] 
(4.1.5.2) 



1.3.0 Are REFERRALS PROCESSED FOR ALL potential special 
education students within the required time lines? 
[E.C. 56321(a)] (4. 1. 5. 5) 

1.3.1 By what procedure can you verify that referrals for 

assessment of need for special education services have 
been appropriately processed? [E.C. 56302] ( 4. 1 . 5. 5 ) 
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1.3.2 What is yoar procedure For a., si. As V . infaf1t:s 
(ages birth to three years ) ; [U iblOO; E.G. 
56301] (4.1.6.1) 

1.3.3 What is the system for transrai 1 1 v :. the re r ^rral to the 
appropriate personnel? [E.C. 5'^G2] (4.1.5. 7 ) 

1.3.4 Is there a handbook or other wf r : on document 
describing the referral/assessmeri process? (E.G. 
56220(a); E.C. 56301] (4.1.0)4 

1.3.5 Is there a procedure for documenting/monitoring 
referral/assessment time lines? (4.1.5.10) 

1.3.6 Who keeps track of the time line that begirr, u P on 
receipt of a referral? [E.C. 56321] (4.1.5..) 

1.3.7 How were you informed that your child was :!erred 
for special education? (4.1.5.6) 

1.3.8 About how long after the referral was made wis 
the IEP meeting held? [E.G. 56344] (4.1.5.10) 

1.3. 9 How long after that did placement occur? 
[E.C. 56344] 

4.0 Are assessment materials arid procedures NONDISCRIM- ^ 
INATORY? [E.C. 56320(a)] (4.1.6) 

1.4.1 Do assessment procedures include provision for 

nondiscriminatory assessment (e.g., sexual, cultural, 
linguistic)? [E.G. 56320(a)] (4.1.6.2) 

5.0 Does the assessment team include PERSONNEL APPROPRIATELY 
TRAINED to administer and interpret test results? 
(E.G. 56320(b)(3)] (4.1.6.2) 

6.0 Are ASSESSMENTS conducted WHEN REQUESTED by the child's 
parent or teacher? [E.G. S6381] (4.1.6.1) 

7.0 Are INDEPENDENT ASSESSMENTS CONSIDERED by the IEP 
team? [E.C. 56329(b)] (4.1.6.7) 

8.0 As a result of referral, is an ASSESSMENT PLAN developed 
within 15 days? [E.C. 56321(a)] (4.1.6) 

1.8.1 How do you use the information on the referral form 
to write the assessment plan? (4.1.6.4) 
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1.9.0 Is PARENT CONSENT obtained before initial asse. .merit? 
[34 CFR 300.504(b)(l)(i) E.G. 56321(a-c)] (4.1.6.5) 

1.9.1 Did you give written consent before the school 
began testing your child? [34 CFR 300. 504(b) ( 1 ) (i )i 
E.C. 56321] (4;h6;5) 

1.9.2 Did you understand what would happen after you 
signed the assessment plan? (E.C. 56321(3)) (4.1.6.5) 

What is done to ensure that the STUDENT IS ASSESSED IN^ 
ALL AREAS related to the suspected disability? [E.C. 
56320(f)] (4.1.6) 



How does the assessment process ensure that NO SINGLE 
PROCEDURE is the sole criterion for determining place- 
ment? [E;C; 56320(e)] (4.1.6) 



Are ASSESSMENTS always COMPLETED and a REPORT WRITTEN 
before the IEP meeting? [E.C. 56320; E.C. 56327] 
(4.1.6.6) 

1.12.1 How and when are you informed of the results of the 
assessment? (4. i. 6. 6) 



Staff Involvement 

1.13.0 Are assessments conducted by a MULTID1 SCI PLIMRY TEAM, 
including a specialist in the area of suspected 
disability? [E.C. 56322(a)] (4.1.6.2) 



1.14.0 Do STAFF UNDERSTAND AND effectively USE the assessment 
process? [300a. 380(b) ; E.C. 56300; E.C. 56301] (4.1.6) 

1.14.1 Tell me about your assessment process. (4.1.6) 



1*15.0 How are assessment RESULTS SHARED with the IEP team 
at or before the meeting? 

1.15.1 How do you ensure that all IEP team members under- 
stand the assessment resu Its when they come to the 
meet irig? (4. 1.6) 



Pa r en t / S t ud an t LnWl 



1.10.0 



1.11.0 



h 16.0 How are PARENTS AND the COMMUNITY informed that they 
CAN MAKE RKKERR&LS? [E.C. 56506(b); E.C. 56302] . 
(4.1.4; 4. 1.5) 



i • i 6 • i How have you been informed that you can make a 

referral for special education? [ E • C . 56506(b); 
E.C. 56301] 

1.17.0 Are PARENTS NOTIFIED that their child has been referred 
for assessment and possible special education services? 
[E.C. 56321(a)] (4.1.5.6; 4.1,6.3) 

1.17.1 Are forms signed oy the parent verifying consent 
available on each child assessed? [E.G. 56321(a); 
E.C. 56321(c)] (4.1.6.5) 

1.17.2 Are the proposed assessment plans in language 
easily understood by the general public, in the 
primary language (or mode of communication) of the 
parent, and do the plans include other required 
components? [E.C* 56321 (a ~c ) ] (4.1.6.5) 

1.18.0 Have parents been informed of all DUE PROCESS RIGHTS, 
including the right to an independent assessment? 
[E.C. 56301; E.C. 56321] (4.1.6.5; 4.8.3) 

1.18.1 How are parents informed of their due process 
rights? [E.C. 56301; E.C* 56321] (4.1.6.5) 

1.18.2 How did the school help you understand your _ legal 
rights? [E.C. 56301; E.C. 56321; E.C. 56329(b)] 
(4.8.3) 



St aff Development 



1.19.0 Have STAFF DEVELOPMENT AND PARENT/ COMMUNITY EDUCATION 

BASED ON NEEDS ASSESSMENTS been provided in the area of 
referral /assessment? (4. 13.3; 4. 13.9) 

Has staff in-service training been held in areas such 
as the right to refer and referral and assessment 
processes? [E.C. 56241(a)] (4. 13.9) 

bid you receive any in-service training on the 
referral/assessment processes ? [E.C. 5624 1 ( a ) (b ) ; 
E.C. 56243 E.C. 56301) (4.13.9) 



1.19. 1 



1. 19.2 



1.19.3 Have you received any in-service training on 
eligibility criteria? (4.1.5.4) 



1.19.4 Has parent /community education been conducted? 
[E.C. 56-220(c)(2); E.C. 56240] (4.1.4) 
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MONITORING OF OTHER STATE AGENCY EDUCATIONAL PROGRAMS 



MINNESOTA 

SUMMARY 

The Minnesota State Department of Education ( DoE ) has 
entered into interagency agreements with .the State Department 
of Corrections (DOC) and the State Department of Public 
Welfare ( DPW ) to carry out the S EA 1 s 9 mandate for general 
supervision of all educational programs for handicapped' 
children in the state. Through these written agreements , DOC 
and DPW agree that the educational programs in the facilities 
which they operate will meet .all the requirements of state and 
federal law and that DoE has the authority to conduct com- 
p 1 i a g monitoring of these facilities, 

OBJECTIVES : 

In entering into agreements with DOC and DPW, the state 
education agency had three bas ic goa Is : 

(1) To improve the o ve ra 1 1 quality of educa t ional ser- 
vices for ha nd i capped students placed in the 
facilities operated by the two agenc ies ; 

(2) To ensure compliance of these programs with state and 
federal laws; and 

(3) To establish an on-going mechan ism to ..identify pro- 
gram def iciencies and a method for their correct ion 
through technical assistance and in-service 
training. 

DEVELOPMENT OF THE POLICY 

Even prior to passage of P.L. 9 4-142, the Minnesota 
Department of Education had been responsible , under state law , 
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for the education of all handicapped children in the state 
( ages 4-21), regardless of agency placement. But while DOC 
and DPW had been meeting the requirements of P. L. 89-7 50 and 
P.L, 8 9-313 respectively^ the Commi ss ipner of Education 
realized that/ to comply with the additional provisions of 
t P;L. 94-142 , more needed to be done. Thus, the federal law 
provided the impetus for DoE to develop interagency agreements 
with both DPW and DOC to address the needs of institu- 
tionalized handicapped children. 

Negotiation of these agreements between DoE and the other 
two state agencies proceeded smoothly. the biggest concerns 
were raised by DOC staff, and these related to the extensive 
new rules surrounding development of an IEP. To resolve this 
arid other issues, operational level staff from both DOC and 
DPW met frequently with their counterparts at DoE to discuss 
the kinds of educational activities that had to be 
strengthened in the institutions, the support needed from DoE, 
and the standards to be included in each of the agreements. 
Supervisory' personnel (Assistant Commissioners) from each 
agency reviewed this work before the agreements were drafted 
and presented to the Commissioners for final approval and 
signature. The DOC agreement was sighed in 1979; the DPW 
agreement in 1980 . 

With these agreements, DoE achieved the necessary degree 
of supervision over all state operated programs. The educa- 
tional programs at the Minnesota Learning Center (operated by 
DPW ) , arid those administered in DOC ' s three facilities that 
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serve youth are. the only programs monitored by SEA staff 
directly. The educational programs of all other state : facili- 
ties are monitored by the appropriate local education agency 
(LEA) because they are operated and staffed with public school 
personnel from the district in which they are geographically 
situated. 

When conducting compliance reviews of the educational 
programs at the correctional facilities and the Minnesota 
Learning Center ( M L C j - 9 the SEA uses basically the same 
procedures it does when conducting LEA compliance mon i tori ng . 
Some changes , as appropriate* are made to the standard form 
and instruments found in the compliance manual developed by 
the Division of Special and Compensatory Education. 

IMPLEMENTATION OF THE POLICIES 

Prior to initiating new monitoring procedures with DOC 
and DPW , the SEA monitoring staff held discuss ions with 
appropr ia te agency per sonne 1 to clarify the standards 
requ i r i ng adherence and prov ide g uidance for responsive 
program design . Then during the 1930-81 school year , the 
first monitoring visits to the three DOC facilities and the 
MLC were conducted . These initial visits did not adhere 
strictly to the SEA ' s compliance mon i tor i ng format ; i ns tead , 
they more closely resembled a technical assistance visit to 
the facilities. With input from staff at* each of the facili- 
ties, program guidelines and plans for implementation of 
program improvements were developed . 
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At the MLC operated by the Department of Public Welfare , • 

the areas that needed improvement to assure compliance with 

P.L. 94-142 were primarily procedural in nature,, e.g. , due 

process and parer>^ rights procedures, team dec i s ion-ritak ing , 

and so forth. Assessment procedures and IEP development also 

>* . . ... 

required som£ attention there. The situation at the DOC 

facilities proved more difficulty hasTc educational programs 

reflecting a continuum of services for handicapped youth 

needed to be established. IEP development required attention 

as well as the more procedural requirements of the law. 

Following the first visit, on-site consultation was provided 

to all, the facilities by DoE staff. Copies of materials with 

positive examples were sent to each agency to illustrate ways 

of correcting procedural deficiencies. Staff were also 

invited to attend the regularly-scheduled LEA in-service 

activities which focused upon skill improvement in a number of 

these areas. 

In February of 1983, the SEA monitoring staff conducted a 
more formal monitoring visit at the MLC (technically, this was 
the second on-site compliance visit to this facility). During 
this visit, positive note was made of the transition work 
being done by MLC outreach'workers, vho travel to LEAs to 
provide them with pertipent data on caildren returning to the 
community. However, the areas of due process, privacy 
protection, and surrogate parents were noted as out of com- 
pliance. (The weakness related to surrogate parents was the 
result of a'change in the state statute.) To assist in 
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correcting these def iQiericies , the SEA has offered on-site 
technical assistance* sent exemplary materials;, and continued 
to invite DPW staff to LEA in-service activities; In short, 
strong qri-going communication exists between the SEA arid MLG 
staff; The SEA monitoring office has approved the Corrective 
Action Plan which was submitted by DPW in response to " the 
February visit and as a result of appropriate foiiow-up 
measures, the MLC is now in compliance. 

The second, more formal compliance visits to the three 
correctional facilities were conducted in the fall of the 
19,8 2-3 3 school year. While much progress was noted, substan- 
tive program deficiencies remained. Full implementation of a 
continuum of services was riot yet apparent in all institutions 
— for example, a functional model was still /feeing used by 
one; The population had so declined in brie facility that there 
was little real overall programming; Another area presenting 
difficulti.es for the correctional facilities was , and con- 
tinues to be, parental permission for assessment. Every time 
a youth is transferred within the correctional system,, the 
permission process must be repeated, and consent from real or 
surrogate parents for assessment is hard to obtain; 

While the Corrective Action Plan submitted by DOC after 
these SEA visits has not yet been approved, the SEA Monitoring 
Office Manager and the DOC Coordinator of Educational 
Programming (who has overall responsibility for the educa- 
tional programs of all the correctional facilities) are 
working together to amel iorate these deficiencies. The SEA 
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provides ori-goirig consultation arid material sharing; 60C 
staff; like DPV7's, are included in relevant hEA in-service 
training activities; Presently, on-site consultation and 
information exchange is being provided to the .Uate 
correctional facility administrators by the SEA. r 

The DPW and DOC facilities aire on a three-year monitoring 
cycle. The follow-up reviews for the DOC facilities are 
scheduled • for fiscal year 1985, although the actual date for 
the return visits will be determined by the amount of agency" 
contact and progress rioted. One area rioted for possible pro- 
cedural change, based on experiences thus far, may be visit 
length. Presently two staff members from the SEA monitoring 
division conduct a one-day visit. Feedback suggests two to 
three days may be necessary. Each of the interagency' agree- 
ments ■•/hifch authorize these SEA monitoring activities is 
reviewed yearly by the appropriate Commissioners , so needed 
changes may be incorporated. 

4 

EFFECTS 

To date, SEA officials have found the overall effect of 
these agreements to be extremely bene f ic ia 1 . For example, 

since implementing the agreements , all teaching personnel y in 

j' . _ ___ _ _ _ ^ * * _ 

the 'four facilities at;e State Board of Education certified. 

Generally, the results from the monitoring activities have 

allowed for more responsive consultation and information 

exchange, and the needs of the facilities are now considered 

by the SEA when developing , L E A and regional in-service 
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training activities. Using a technical assistance approach, a 
method for problem identification arid resolution has been 
implemented. Many deficiencies have been corrected arid the 
wbrkihg relationships how established are being used for joint 
problem solving where weaknesses remain. 
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